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EXECUTIVE SUMMARY 


Executive Summary 

About the Bill & Melinda Gates Foundation’s Early College 
High School Initiative (ECHSI) 

In 2002, the Bill & Melinda Gates Foundation launched the ECHSI with the primary goal of 
increasing the opportunity for underserved students to earn a postsecondary credential. To 
achieve this goal, Early Colleges provide underserved students with exposure to, and 
support in, college while they are in high school. Early Colleges partner with colleges and 
universities to offer all students an opportunity to earn an associate’s degree or up to two 
years of college credits toward a bachelor’s degree during high school at no or low cost to 
the students. The underlying assumption is that engaging underrepresented students in a 
rigorous high school curriculum tied to the incentive of earning college credit will motivate 
them and increase their access to additional postsecondary education and credentials after 
high school. Since 2002, more than 240 Early Colleges have opened nationwide. 

About the Study 

This study focused on the impact of Early Colleges. It addressed two questions: 

1. Do Early College students have better outcomes than they would have had at other 
high schools? 

2. Does the impact of Early Colleges vary by student background characteristics (e.g., 
gender and family income)? 

To answer these questions, we conducted a lottery-based randomized experiment, taking 
advantage of the fact that some Early Colleges used lotteries in their admissions processes. 
By comparing the outcomes for students who participated in admissions lotteries and were 
offered enrollment with the outcomes for students who participated in the lotteries but were 
not offered enrollment, we can draw causal conclusions about the impact of Early Colleges. 

The primary student outcomes for this study were high school graduation, college 
enrollment, and college degree attainment. We also examined students’ high school and 
college experiences. Data on student background characteristics and high school outcomes 
came from administrative records from schools, districts, and states; data on college 
outcomes came from the National Student Clearinghouse (NSC); and data on high school 
and college experiences and intermediate outcomes such as college credit accrual came 
from a student survey. 
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EXECUTIVE SUMMARY 


We assessed the impact of Early Colleges on these outcomes for a sample of 10 Early 
Colleges that did the following: 

■ Enrolled students in grades 9-12 and had high school graduates in the study years 
(2005-2011) 

■ Used lotteries as part of the admission processes in at least one of the study cohorts 
(students who entered ninth grade in 2005-06, 2006-07, or 2007-08) 

■ Retained the lottery records 

Eight of the 10 Early Colleges in the study were included in the student survey. The overall 
study sample included 2,458 students and the survey sample included 1,294 students. The 
study extended through three years past high school. 

Key Study Findings 

■ High School Graduation 

Early College students were significantly more likely to graduate from high 
school than comparison students. Overall, graduation rates for both groups were 
high. However, 86 percent of Early College students graduated from high school, 
which was significantly higher than the 81 percent for comparison students. 

■ College Enrollment 

Early College students were significantly more likely to enroll in college than 
comparison students. During the study period, 80 percent of Early College students 
enrolled, compared with 71 percent of comparison students. In addition, Early College 
students were more likely than comparison students to enroll in both two-year and 
four-year colleges or universities. Although the gap in enrollment rates between the 
two groups decreased over time, comparison students’ college enrollments did not 
catch up to those of Early College students during the study period. 

■ College Degree Attainment 

Early College students were significantly more likely to earn a college degree 
than comparison students. During the study period, 22 percent of Early College 
students earned a college degree (typically an associate’s degree), as compared 
with only 2 percent of comparison students. In addition, Early College students 
earned college degrees far earlier than is typical: 20 percent of Early College 
students earned a degree by the time they graduated from high school. 
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EXECUTIVE SUMMARY 


■ Impact for Student Subgroups 

Early College impact generally did not differ by subgroup, and when the 
impact differed, the difference was generally in favor of underrepresented 
groups. The Early College impact on high school graduation and college enrollment 
did not differ significantly based on gender, race/ethnicity, family income, first- 
generation college-going status, or pre-high school achievement. In other words, all 
student groups experienced the impact of attending an Early College. The Early 
College impact on college degree attainment did not differ based on first-generation 
college-going status, but were stronger for female than male students, stronger for 
minority than non-minority students, stronger for lower income than higher income 
students, and stronger for students with higher middle school achievement than 
lower achieving students. 

Summing Up 

Although the findings from this study are applicable only to the 10 Early Colleges included in 
the study sample, they provide strong evidence for the positive impact of Early Colleges on 
students. Early College students had a greater opportunity than their peers to enroll in and 
graduate from college. They also appeared to be on a different academic trajectory, with 
Early College students earning college degrees and enrolling in four-year institutions at 
higher rates than comparison students. In addition, Early Colleges appeared to mitigate the 
traditional educational attainment gaps between advantaged and disadvantaged students. 

These Early Colleges likely attracted academically prepared and ambitious students to their 
lotteries. Prior to high school, lottery applicants generally performed above the state average 
in the ELA and mathematics assessments. Furthermore, both Early College and comparison 
students had impressive high school graduation rates and college enrollment rates, though 
Early College students’ rates were higher. 

The Early Colleges in this study yielded significant and meaningful improvements in almost 
every student outcome examined. Early College students were benefitting from their Early 
College experience beyond high school, and we expect these benefits to continue. For 
example, Early College students may earn more college degrees, may accrue less 
educational debt, and may begin their careers sooner (and thus may have higher lifetime 
earnings) compared with other students. 
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CHAPTER 1. INTRODUCTION 


Chapter 1. Introduction 

The Early College High School Initiative (ECHSI), launched by the Bill & Melinda Gates 
Foundation in 2002, provided funds for the development of Early College High Schools 
(hereafter referred to as “Early Colleges”). Early Colleges offer students who are traditionally 
underrepresented in postsecondary education the opportunity to pursue a high school diploma 
while simultaneously earning college credits. The primary goal of the ECHSI is to increase 
students’ access to a postsecondary credential. The solution offered by the ECHSI is to 
improve underrepresented students’ likelihood of earning a college degree by enrolling them 
in college courses while they are in high school and can receive support from high school 
staff. 

This report provides the findings from an evaluation of the ECHSI. The evaluation sought to 
determine if Early College students had better outcomes than they would have had at other 
high schools. We used a lottery-based randomized experiment. The primary student 
outcomes evaluated were high school graduation, college enrollment, and college degree 
attainment. 

There is substantial evidence that a postsecondary degree or credential prepares students 
for successful entry into the workforce. Bachelor’s degree holders earn more over a lifetime 
than individuals with only a high school diploma (Carnevale, Rose, & Cheah, 2011), and 
college degree earners fared better in the recent American recession than adults who held 
only a high school diploma (Grusky, Bird, Rodriguez, & Wimer, 2013). Moreover, workforce 
projections consistently predict that the lion’s share of future jobs will require a 
postsecondary degree (Carnevale, Smith, Stone, Kotamraju, Steuernagel, & Green, 2011). 
Postsecondary success therefore represents the most critical goal for the initiative. 

The ECHSI focuses in particular on supporting underrepresented students in achieving a 
college credential. Research has consistently shown that minority students and students 
from disadvantaged families are underrepresented as college degree recipients. First, 
Hispanic and African American students are less likely than non-minority students to earn a 
bachelor’s degree (NCES, 2012b). Second, low-income students are less likely to earn a 
college degree of any type than higher income students (NCES, 2012c). Third, students who 
are the first in their family to go to college are less likely to leave college with a degree than 
students whose parents have attended college (NCES, 2012c). As a result, student 
subgroups historically underrepresented among college degree recipients are the intended 
target population of the ECHSI. 

Offering college courses to high school students is not unique to the ECHSI. In 2010-11, 

82 percent of public high schools offered dual credit courses (Thomas, Marken, Gray, & 
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Lewis, 2013). In addition, approximately 53 percent of postsecondary institutions reported 
enrolling high school students in college courses either within or outside formal dual 
enrollment programs (Marken, Gray, & Lewis, 2013). 1 Despite the availability of dual 
enrollment programs, less than 10 percent of public high school students took advantage of 
them in 2010-11. 2 

Early Colleges facilitate dual enrollment through established course sequences. Through the 
ECHSI, Early Colleges partner with colleges and universities to offer enrolled students an 
opportunity to earn an associate’s degree or up to two years of college credits toward the 
bachelor’s degree during high school at no cost to the students. The underlying assumption 
is that engaging underrepresented students in a rigorous high school curriculum tied to the 
incentive of earning college credits will motivate them and increase their access to additional 
postsecondary education and credentials after high school. 

Early Colleges differ from traditional high schools in more than just offering dual enrollment. 
These differences are codified in the five Core Principles of the ECHSI: 

1. “Early college schools are committed to serving students underrepresented in higher 
education.” 

2. “Early college schools are created and sustained by a local education agency, a 
higher education institution, and the community, all of whom are jointly accountable 
for student success.” 

3. “Early college schools and their higher education partners and community jointly 
develop an integrated academic program so all students earn one to two years of 
transferable college credit leading to college completion.” 

4. “Early college schools engage all students in a comprehensive support system that 
develops academic and social skills as well as the behaviors and conditions 
necessary for college completion.” 

5. “Early college schools and their higher education and community partners work with 
intermediaries to create conditions and advocate for supportive policies that advance 
the early college movement” (Jobs for the Future, 2008, p. 2). 

Of particular note is the initiative’s focus on supports as part of Core Principle 4. Early 
Colleges strive to have all students, not just academically advanced students, succeed in 
college courses. To accomplish this goal, they provide additional supports to students for 
high school and college courses. Early Colleges offer advisories, classes with high school 

1 Dual enrollment programs are formal mechanisms that enable high school students to take college courses. Many dual 
enrollment programs are also dual credit programs, in which students can earn both high school and college credits for the 
same course (Kleiner & Lewis, 2005). We use the term “dual enrollment” hereafter to encompass both types of programs. 

2 Calculated based on the total number of students enrolled in college during high school (Thomas et al., 2013) divided by the 
total number of students in high school (NCES, 2012a). 
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teachers to support particular college courses, and more accessible individual supports (AIR 
& SRI, 2008, 2009). 3 In addition, Early Colleges provide supports in the formal transition to 
college. Specific supports, such as help in completing college applications and financial aid 
forms, are important given that the complexity of the process is a barrier to college 
attendance for academically qualified, low-income students (Bettinger, Long, Oreopoulos, & 
Sanbonmatsu, 2009; Hoxby & Avery, 2012). 

In addition to the Core Principles, the ECHSI emphasizes the 3 Rs—rigor, relevance, and 
relationships. Rigorous instruction is intended to build students’ content knowledge and 
learning habits. Relevant instruction is designed to engage students in understanding why 
they are learning a topic and in making real-world connections. Finally, personalized 
relationships between instructors and students are expected to support student engagement 
and achievement. Taken together, these principles and attributes of the ECHSI cover the 
features recommended for improving high school students’ access to and success in college 
(Tierney, Bailey, Constantine, Finkelstein, & Hurd, 2009). 

Since 2002, more than 240 Early Colleges have opened nationwide as part of the ECHSI 
(Jobs for the Future, 2013). The ECHSI operates through 13 grantee organizations, or 
intermediaries, that receive foundation funding to work with local partners—such as school 
districts, community organizations, tribes, high schools, community colleges, and 
universities—to open Early Colleges. 4 

In the 11 years since the ECHSI started, the call to improve high school education, college 
access, and college degree attainment has become louder. In the 2013 State of the Union 
Address (Office of the Press Secretary, 2013), President Obama stated, “Now at schools like 
P-Tech [an Early College] in Brooklyn, a collaboration between New York Public Schools and 
City University of New York and IBM, students will graduate with a high school diploma and an 
associate's degree in computers or engineering. We need to give every American student 
opportunities like this.” The ECHSI strives to provide just these opportunities; the findings from 
this report can inform policymakers and stakeholders about the ECHSI, its schools, and its 
impact on students. 

Research on Dual Enrollment and Early Colleges 

Although the AIR and SRI evaluation has been the most comprehensive study of the 
ECHSI, there have been other evaluations of dual enrollment and Early Colleges. Below, we 
review other descriptive and correlational studies of dual enrollment and Early Colleges as 


3 The purpose of an advisory class is to ensure that at least one adult in the school gives attention to the academic and 
emotional needs of each student. 

4 Organizations other than the 13 supported by the foundation have opened Early Colleges. For example, the state of Michigan 
has started Early Colleges (Floffman & Webb, 2009). 
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well as several studies of the impact of high school reform initiatives, including the ECHSI, 
on student outcomes. 

Dual Enrollment 

The primary goal of providing college experiences to high school students is to increase the 
likelihood that students will finish a postsecondary credential. Correlational and quasi- 
experimental research provides suggestive evidence of dual enrollment’s effectiveness in 
meeting goals such as college preparedness and completion. A correlational study 
conducted in two states (Karp, Calcagno, Hughes, Jeong, & Bailey, 2007) found that dual 
enrollment was associated with positive outcomes on a number of measures: Students who 
had taken college classes during high school were more likely to earn high school degrees, 
enroll in college, enroll in a four-year college, enroll full time, and persist in college than 
were students without college experience. In addition, the study found that students who 
received college credits during high school had higher college grade point averages (GPAs) 
and earned more college credits within three years of high school graduation. Another 
correlational study using a large federal database found similar positive outcomes for dual 
enrollment students, including enrollment in college, persistence in college, and college 
graduation (Swanson, 2008). A quasi-experimental study using a large federal database 
also found that dual enrollment participation increased the probability of attaining any 
postsecondary degree by 8 percent and a bachelor’s degree by 7 percent (An, 2012). A 
quasi-experimental study of dual enrollment in Texas found that participation was 
associated with college attendance and completion (Struhl & Vargas, 2012). 

Dual enrollment has also been associated with positive outcomes for students traditionally 
underrepresented in college. A correlational study found that dual enrollment was 
associated with gains in college enrollment and GPA for low-income students and lower 
achieving students in Florida (Community College Research Center, 2012). A quasi- 
experimental study on the Concurrent Course Initiative (CCI) in California, which 
implements career-focused dual enrollment and targets students who are low income, 
struggling academically, and traditionally underrepresented in college, found that 
participants had higher graduation rates, were more likely to enroll in a four-year college, 
had greater college persistence rates, accumulated more college credits as they progressed 
through college, and were less likely to enroll in basic skills courses in college than non¬ 
participants (Hughes, Rodriguez, Edwards, & Belfield, 2012; Rodriguez, Hughes, & Belfield, 
2012). Similarly, a quasi-experimental study using national longitudinal data found that first- 
generation students who participated in dual enrollment were more likely to attain a 
postsecondary degree (of any kind) and earn a bachelor’s degree (An, 2012). 
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Several high school programs integrate college-level content or provide college-course access 
to high school students. A descriptive study of programs that allow high school students to 
take college-level classes for college credits, such as tech-prep programs, International 
Baccalaureate programs, and middle college high schools, found three primary benefits for 
students: (a) an opportunity to earn free college credit, (b) a chance to gain “a taste” of 
college, and (c) increased student confidence in their own academic abilities (Hughes, Karp, 
Fermin, & Bailey, 2005). A correlational study (Speroni, 2011) comparing outcomes for 
students participating in dual enrollment with those for students taking Advanced Placement 
(AP) courses found that dual enrollment students were more likely to go to college after high 
school than students who took AP courses. However, the positive outcomes for dual 
enrollment were present only for students who took those courses on a college campus. 

Early Colleges 

Although the research on dual enrollment is promising, Early Colleges provide a more 
comprehensive experience than just dual enrollment, as stated in the Core Principles. A 
descriptive study conducted by the National Center for Restructuring Schools and Teaching, 
which works with the Middle College National Consortium, an ECHSI grantee organization, 
found that Middle Colleges succeeded in providing their students with early access to 
college courses, and that each year higher numbers of students from diverse racial, ethnic, 
and socioeconomic backgrounds participated in college coursework (Kim & Barnett, 2008). 

In addition, a study of Early Colleges in Texas found that Early College attendance was 
associated with a higher probability of exceeding state standards in mathematics in 9th, 

10th, and 11th grades (SRI, 2011). 

As the number of Early Colleges in operation long enough to graduate their first classes of 
students has grown, data are becoming available on post-high-school outcomes. In a 
descriptive study that examined the outcomes for 900 graduates from 11 Early Colleges in 
Texas, students accrued an average of 24 college credits by the time they graduated, and 
more than one third earned an associate’s degree (Jobs for the Future, 2011). Webb and 
Mayka (2011) examined graduate data for the 2007, 2008, and 2009 graduating classes in 
64 Early Colleges. They reported that 24 percent of 2009 graduates who had enrolled in 
Early College for four years earned an associate’s degree or two years of college credit, and 
44 percent accrued at least one year of college credit by high school graduation. Nearly 
three-fourths (73 percent) of the graduates, for whom data were available, enrolled in 
college the year after high school. In addition, a longitudinal qualitative study of two Early 
Colleges, which followed students from ninth grade through their second year of college, 
found positive social outcomes for Early College students, including improvements in their 
ability to adapt to college demands and increases in the likelihood of them seeking out 
academic support, taking on leadership roles, and supporting their peers (Nakkula, 2011). 
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These Early College studies are encouraging, but none compared Early College students 
with non-Early College students. One correlational study that did compare Early College and 
non-Early College students, conducted by AIR and SRI (2009), found that students in Early 
Colleges graduated at higher rates than students in other schools in the same district. 

Impact Studies 

The studies described thus far cannot attribute the differences in student outcomes to the 
Early Colleges. In fact, given the rigorous curriculum involved, it could be that Early Colleges 
attract, recruit, and enroll students who are different from students attending typical high 
schools. For example, Early College students could be more motivated, be better prepared, 
and have more supportive families. In other words, the previously cited correlational studies 
do not provide a valid assessment of the impact of Early Colleges. 

The impact of Early College is best determined through an experiment in which students are 
assigned to Early Colleges or other high schools through a randomization process such as a 
lottery. The use of random assignment ensures that students in the treatment (Early 
College) group and their counterparts in the comparison group are similar in both observed 
and unobserved characteristics prior to high school, meaning that subsequent differences in 
student outcomes can be attributed to the different types of high schools they attended 
rather than to preexisting differences in motivation, aptitude, educational aspirations, or 
other characteristics. 

One experimental study of Early Colleges is a longitudinal study of the ECHSI’s North 
Carolina Learn and Earn Initiative that is currently being conducted by the SERVE Center. 
The study found that Early College students were more likely to be “on track for college” 
than comparison group students (SERVE Center, 2010). Early College ninth grade students 
were more likely to take core college preparatory courses and succeed in them. Early 
College students also had higher attendance rates, lower suspension rates, and higher 
levels of engagement than control students (Edmunds, Bernstein, Unlu, Glennie, & Willse, 
2012; Edmunds, Willse, Arshavsky, & Dallas, 2013). 

Two impact studies of other high school reform initiatives similar to the ECHSI yielded 
positive findings. MDRC used a random assignment design to study the impact of Career 
Academies in nine high schools. MDRC tracked students for 12 years from the time of high 
school entry. They found that Career Academies improved the likelihood that students at 
risk of dropping out stayed in school, made academic progress, and graduated (Kemple & 
Snipes, 2000). In addition, Career Academies increased earnings for young men and 
students at risk of dropping out of high school (Kemple, 2004, 2008). 
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The second study, by Mathematica Policy Research, examined a predecessor of the ECHSI, 
a middle college high school located on a college campus, which operated as an alternative 
school for dropout students or students with poor attendance (Dynarski, Gleason, 
Rangarajan, & Wood, 1998). The study used a random assignment design with a sample of 
394 students. Most of the students were over age for their grade or had previously dropped 
out of school. Most comparison group students participated in other regular or alternative 
education programs. The study found that students at this middle college high school were 
no more likely to stay in school or graduate in the two years after random assignment than 
students in the comparison group. 

The ECHSI Evaluation 

The results reported in the chapters that follow are based on work conducted during the final 
years of a multi-year evaluation of the ECHSI conducted by AIR and SRI. In 2002, AIR 
initiated an evaluation of the ECHSI with support from the Bill & Melinda Gates Foundation. 
The original evaluation was designed to answer three primary research questions: 

1. What are the demographic, structural, organizational, and instructional 
characteristics of Early Colleges? 

2. What factors support or inhibit the planning and development of Early Colleges? 

3. What are the intermediate and long-term outcomes for students attending Early 
Colleges, especially for students traditionally underrepresented in the postsecondary 
system? 

Between 2002 and 2009, we interviewed each of the ECHSI grantees annually, surveyed 
school leaders in every Early College annually, conducted more than 70 site visits, and 
surveyed more than 5,000 students in more than 70 Early Colleges. Drawing on these data, 
AIR and SRI produced annual reports describing the characteristics of Early Colleges and 
the ECHSI participants, describing factors facilitating and constraining the planning and 
development of Early Colleges, and examining student outcomes. 

Findings from the first eight evaluation years indicated that Early Colleges were generally 
implemented with fidelity to the Core Principles and had positive student outcomes. 5 Over half of 
the Early Colleges were located on college campuses. To help students succeed in their 
coursework and the transition to college, Early Colleges provided a variety of supports, including 
advisories, tutoring, and academic support classes. The Early College student population was 
67 percent minority and 53 percent low income—both higher percentages than found in the 
districts the students came from. Additionally, 46 percent of Early College students were first- 


5 Results are from AIR & SRI, 2005, 2006, 2007, 2008, and 2009. 
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generation college students. Academically, Early Colleges boasted higher proficiency rates on 
state achievement tests than comparison districts. Most students took advantage of the 
opportunity to earn college credit, with 61 percent of students reporting taking at least one 
college class (including 73 percent of 12th graders) and a college GPA of 3.1. On average, Early 
College students graduated with a semester to a year of college credits, and 88 percent of 
graduates enrolled in college in the fall after graduation. 

Overview of the ECHSI Impact Study 

In 2010, the AIR-SRI evaluation of the ECHSI moved in a new direction, focusing on 
determining the impact of Early Colleges rather than documenting the development and 
implementation of the ECHSI. The overarching research question for the impact study is the 
following: 

■ Do Early College students have better outcomes than they would have had at other 
high schools? 

In addition to the overall impact of Early College, the study also examines potential variation 
in Early College impact: 

■ Does the impact of Early Colleges vary by student background characteristics (e.g., 
gender and family income)? 

We addressed these questions using a lottery-based randomized experiment, taking 
advantage of the fact that starting with the second year of the ECHSI, some of the Early 
Colleges used lotteries in their admissions processes. Our descriptive studies found 
consistently positive experiences and outcomes for Early College students, but we could not 
determine whether the positive findings were attributable to the characteristics of the 
students who chose to attend Early Colleges or to the Early Colleges themselves. In a 
lottery study, the only measurable difference between lottery applicants who were offered 
enrollment (“won the lottery”) and lottery applicants who were not offered enrollment (“lost 
the lottery”) is the lottery itself. By comparing outcomes for students who were offered 
enrollment with those who were not offered enrollment, we can draw valid causal 
conclusions about the impact of Early Colleges. 

The primary student outcomes for this study are consistent with the goals of the ECHSI and 
the foundation’s college-readiness and postsecondary success strategies (Bill & Melinda 
Gates Foundation, 2009). The outcomes include whether students 

■ graduated from high school, 

■ enrolled in college, and 
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CHAPTER 1. INTRODUCTION 


■ earned a college certificate or degree (either during their time in high school or 

afterward). 

We assessed the impact of Early Colleges on these outcomes and other student outcomes 
based on a sample of 10 Early Colleges that (1) enrolled students in grades 9-12 and had 
high school graduates in the study years (2005-2011), (2) used lotteries in their admission 
processes in at least one of the study cohorts: ninth graders in 2005-06, 2006-07, or 2007- 
OS, and (3) retained the lottery records. This study followed students from ninth grade 
through up to four years after high school. Exhibit 1.1 displays the progression of students 
through high school for each cohort, if the students progressed at the expected pace. Study 
cohorts are numbered based on the year in which students entered ninth grade: 

■ Cohort 1 students entered ninth grade in 2005-06 and were expected to graduate in 

2008- 09 if they progressed at the normal pace. 

■ Cohort 2 students entered ninth grade in 2006-07 and were expected to graduate in 

2009- 10. 

■ Cohort 3 students entered ninth grade in 2007-08 and were expected to graduate in 

2010 - 11 . 


Exhibit 1.1. Study Cohorts and Students’ Expected Progression 


Cohort 

2005-06 

2006-07 

2007-08 

2008-09 

2009-10 

2010-11 

2011-12 

2012-13 

1 

Year 1 

Year 2 

Year 3 

Year 4 

Year 5 

Year 6 

Year 7 

Year 8 


9th 

10th 

11th 

12th 

1st 

post-HS 

2nd 

post-HS 

3rd 

post-HS 

4th 

post-HS 

2 


Year 1 

Year 2 

Year 3 

Year 4 

Year 5 

Year 6 

Year 7 



9th 

10th 

11th 

12th 

1st 

post-HS 

2nd 

post-HS 

3rd 

post-HS 

3 



Year 1 

Year 2 

Year 3 

Year 4 

Year 5 

Year 6 




9th 

10th 

11th 

12th 

1st 

post-HS 

2nd 

post-HS 


/VOTE: This table displays a traditional educational progression. Students may take more or less time to complete high school. 


We drew on student record data from schools, districts, states, and the National Student 
Clearinghouse (NSC), 6 as well as from a student survey to measure student outcomes. The 
high school student record data cover student high school outcomes through spring 2011. 


6 The National Student Clearinghouse (NSC) collects data from higher education institutions across the country. Data are 
collected on student enrollments and degree completions. The data cover over 96 percent of all student enrollments in public 
and private colleges and universities (2013). 
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CHAPTER 1. INTRODUCTION 


The NSC data cover student college outcomes through fall 2012. 7 The student survey was 
administered October 2011 through January 2012. Therefore, for the youngest students 
(Cohort 3), all data collections capture students at least through their expected high school 
graduation date and into their expected first year of college enrollment. 

The remaining sections of this report are organized as follows. Chapter 2 provides a 
detailed description of the study design. Chapter 3 reports the findings on high school and 
college outcomes, and Chapter 4 reports the findings on student high school experiences. 
Chapter 5 summarizes the study’s findings. The technical appendices provide additional 
information on study design, data analyses, and findings. Appendix A defines the measures 
and identifies the data sources. Appendix B describes the administrative and survey data 
samples; it includes study counts, data collection procedures, and survey response rates. 
Appendix C displays a table of missing data rates for administrative and survey measures. 
Appendix D explains the technical details of the impact models that were used to respond to 
the two research questions. Appendix E shows analysis statistics for all impact findings 
presented in the report. Appendix F presents a series of sensitivity analyses. Appendix G 
shows the results of an analysis of the impact that Early Colleges had on students who 
attended Early College. 


7 Note that the only data available for the 2012-13 year were the fall enrollments from the NSC. We did not have enrollment or 
degree attainment data for the full academic year. 
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Chapter 2. Research Design 

This chapter provides a description of the conceptual framework as well as an overview of 
the study samples, data collection activities, and analytic approach. Additional details on the 
research design are included in the technical appendices. 

ECHSI Impact Study Conceptual Framework 

To guide the study, we formulated a conceptual framework describing the key features of 
students’ Early College experiences that are hypothesized to influence their subsequent 
high school and postsecondary outcomes and ultimate educational attainment (see Exhibit 
2.1). Early Colleges are expected to implement programs consistent with the ECHSI Core 
Principles. In particular, they are expected to provide college exposure, rigorous academics, 
and student supports. These high school features in turn are expected to promote improved 
high school outcomes, including high school achievement and graduation. Students’ high 
school outcomes may lead them to engage in further college education or lead directly to 
college degree attainment if the students complete sufficient postsecondary credits while in 
high school. Finally, student background characteristics may affect student outcomes 
differently both during and after high school, and may also serve as potential moderators for 
the Early College impact on outcomes. 

We measured several key aspects of students’ experiences in high school, high school 
outcomes, college outcomes, and educational attainment. Here, we provide an overview of 
the measured variables. Appendix A lists all of the measured variables, together with 
variable descriptions and the data sources. 
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Exhibit 2.1. ECHSI Impact Study Conceptual Framework 


ECHSI Core 
Principles 


| Early College 
/ Experiences 

College 

Exposure 

Rigorous 

Academics 

Supports 


High School 
^ Outcomes 


Achievement 


Graduation 




College 

Outcomes 

Enrollment 

Performance 



Educational 

Attainment 


College Degree 



Student Background Characteristics 


To measure students’ Early College experiences, we focused on features intended to 
differentiate Early Colleges from traditional high schools: College Exposure , Rigorous 
Academics, and Supports. To measure college exposure, we examined students’ college 
enrollment and credit accumulation while in high school. To measure rigorous academics, we 
examined course sequences in mathematics and science and frequency of rigorous instructional 
activities. To measure supports, we examined instructor support for students’ academic, social, 
and emotional well-being; the college-going culture in the school; and support for the college 
enrollment process. We focused on college preparation-specific supports because research 
suggests that supports in college preparation are associated with higher college enrollment and 
completion rates for underrepresented students (Bettinger, Long, Oreopoulos, & 

Sanbonmatsu, 2009; Hoxby & Avery, 2012). 8 

High School Outcomes provide stepping stones to the initiative goal of college degree 
attainment. We examined two high school outcomes: Achievement and Graduation. To 
measure achievement, we examined high school ELA and mathematics assessment scores 


8 Other features may also differ between Early Colleges and the schools that comparison students attended. For example, 
some Early Colleges have a STEM focus. Additionally, there may be other ancillary differences. For example, students in Early 
Colleges may benefit from attending school with like-minded peers. On the other hand, there may be some opportunity costs 
associated with being in an Early College. For example, students in Early Colleges may have less access to extracurricular 
activities or may face more challenges associated with transportation than students in other schools. 
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and grade point averages. For high school graduation, we measured the percentage of 
students earning a high school diploma. 

Because students may not attend college after high school, we differentiate between college 
experience while in high school and college experience after high school. College Outcomes 
include outcomes that would be precursors to a college degree if students continued their 
education after high school: Enrollment and Performance. To measure Enrollment, we 
examined whether or not students enrolled in college and the type of institution they 
attended. To measure Performance, we examined developmental course-taking, college 
GPA, educational plans, students’ perceptions of academic difficulty in college, and 
persistence (students’ year-to-year college enrollment). 

To measure Educational Attainment, we examined whether students earned a College 
Degree, either during or after high school. By college degree, we mean any postsecondary 
credential, including certificates, associate’s degrees, or bachelor’s degrees. 9 

To measure Student Background Characteristics, we examined the following student 
variables, which may influence student outcomes both during and after high school: gender, 
student race and ethnicity, first-generation college-going status, low-income status, and pre- 
high-school academic achievement. 

Identifying Eligible Early Colleges 

To identify Early Colleges that were eligible for the impact study, we sought Early Colleges that 
met several criteria. Generally, the broad criteria were that the school was open and had 
graduates during the study period, used a lottery for admissions, and retained lottery records. 

We examined the 154 Early Colleges that were open by fall 2007 to identify eligible sites for the 
study, and we included 10 sites in the final sample. Exhibit 2.2 shows a summary of the reasons 
that Early Colleges were excluded and the number of Early Colleges excluded for each reason. 


9 We use the term “degree,” though it includes certificates, for clarity and to emphasize the most commonly attained 
credentials, which were postsecondary degrees. As will be presented in Chapter 3, few students earned a certificate. 
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Exhibit 2.2. Flowchart for Identifying Early Colleges for the ECHSI Impact Study 



Yes 




n = 123 


Used a lottery 
for admissions 


No 




Not oversubscribed. Early College 
or district selected students 


n = 103 


No 


n = 10 


Misplaced or destroyed records 


Early College in study 
n = 10 
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Using Lotteries Retrospectively 

The study design is retrospective: we sought sites that conducted lotteries long enough in 
the past that students who had entered Early Colleges at the time the lotteries were 
conducted would have had the opportunity to graduate from high school and enter 
postsecondary education by the time data collection concluded. The retrospective feature of 
the design made it possible to estimate the impact on postsecondary outcomes within a 
reasonably short time frame. 

To be included in the study sample, we sought Early Colleges that held a lottery for at least 
one of the three study cohorts shown in Exhibit 1.1. Three Early Colleges in the sample 
conducted lotteries for Cohort 1 (2005-06); seven conducted lotteries for Cohort 2 (2006- 
07); and seven conducted lotteries for Cohort 3 (2007-08). Appendix B shows the cohorts 
included for each Early College. The lotteries that were the basis for the study were 
conducted by school staff, district staff, or contractors. Because we did not have control over 
the random assignment process, we took steps to examine the quality of the lotteries. We 
put together a detailed account of the lottery process for each Early College, drawing on 
information obtained from extensive conversations with staff members in each school. In 
some cases—for example, sites where fellow researchers ran the lottery—we are quite 
confident in the quality of the lottery process. In other cases—for example, sites where the 
current school or district staff were not present at the time of the lottery—we have less 
information. Appendix B includes more details about lottery documentation, and Appendix F 
includes sensitivity analyses where we excluded sites with unclear lottery documentation. 

There is some precedent for working with historical lottery records. A retrospective lottery 
study funded by the Bill & Melinda Gates Foundation investigated the impact of New York 
City’s small schools of choice on students’ academic achievement (Bloom, Thompson, & 
Unterman, 2010). The researchers relied on the districtwide lottery records for school 
enrollment, which resulted in a sample of more than 21,000 students in four annual cohorts 
who applied to 105 oversubscribed schools. A different study of New York City charter schools 
used 725 lotteries (one for each grade in each school each year) to estimate the effect of 
charter schools’ policies on achievement (Hoxby & Murarka, 2009). Others successfully used 
individual charter schools’ historical records of their lotteries in a retrospective lottery design to 
study the effectiveness of charter schools in Boston (Abdulkadiroglu et al., 2009) and 
Massachusetts (Angrist et al., 2011). Both studies included high schools and found that the 
treatment and comparison groups did not differ significantly on background characteristics or 
baseline test scores, which suggests that the lotteries held by the schools were valid. 
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Identifying Early College and Comparison Students 

Early Colleges provided the following information about the status of each student who 
applied: 

■ Whether the student was a lottery participant 

■ Whether the student was accepted from the initial lottery or from a randomized wait¬ 
list 

In our study, we defined “Early College students” (or treatment students) as all students who 
were offered admission to the Early College from a lottery prior to the first day of school. We 
did not differentiate between students who were accepted from the initial lottery and 
students who were accepted from a randomized wait-list. Our comparison students included 
all students who participated in the lottery but were not offered admission to the Early 
College prior to the first day of school. Although some Early Colleges held lotteries to accept 
students in multiple grades, our study included only the lotteries for the first grade level of 
the high school. Exhibit 2.3 includes the number of sites and Early College and comparison 
students for each cohort year. The 10 study Early Colleges held a total of 17 lotteries across 
cohorts. 


Exhibit 2.3. Number of Early Colleges and Number of Students in Study Sample, by Cohort Year 


Cohort Year 

Number of Early 
Colleges 

Number of 

Early College Students 

Number of 

Comparison Students 

2005-06 

3 

133 

343 

2006-07 

7 

431 

551 

2007-08 

7 

480 

520 

TOTAL 

10* 

1,044 

1,414 


* Some Early Colleges had multiple participating cohorts; therefore the number of Early Colleges across the three cohorts is 
more than the total number of Early Colleges in the study sample. 


Random assignment to the Early College or comparison condition is not the same as attending 
or not attending an Early College. Early College students may not have attended an Early 
College (i.e., “no-shows”), and comparison students may have attended an Early College (i.e., 
“crossovers”). We defined a “no-show” as an Early College student who did not enroll in the 
Early College during the first year of high school, and a “crossover” as a comparison student 
who attended an Early College during the first year of high school. Comparison students who 
attended an Early College that was not in our study sample were also considered “crossovers,” 
as they received “treatment” similar to that for the Early College students. Across the 10 study 
sites, the no-show rate was 22 percent and the crossover rate was 2 percent (see Appendix B 
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for no-show and crossover rates for each site). 10 One site had a sizable no-show rate (close to 
or over 50 percent in three lotteries), and one site had a sizable crossover rate (28 percent). We 
conducted sensitivity analyses with and without these sites on several key outcomes (see 
Appendix F for results). 

Characteristics of Study Schools 

Before delving into the characteristics of Early College and comparison students, we first 
examine the schools that study participants attended: the 10 Early Colleges in the study and 
the schools that comparison students attended. Drawing on data from an Early College 
survey (see details about the survey in AIR & SRI, 2009) and publicly available data, we 
examined the characteristics of these schools in 2007-08, when students in the youngest 
cohort were in 9th grade and students in the oldest cohort were in 11th grade. * 11 Below, we 
first describe the Early Colleges in our study sample, followed by the schools attended by 
the comparison students. 

Early Colleges 

The 10 Early Colleges were located in five states throughout the country: 5 in urban areas, 2 
in mid-sized cities, and 3 in small towns. Nine of the Early Colleges opened as new schools 
and one was an existing school that became an Early College. The average total student 
enrollment was about 290, ranging in size from about 100 to over 600. All of the schools met 
the typical definition of a small school (less than 150 students per grade), with about 90 
students per grade on average. In 2007-08, four recently opened Early Colleges were in the 
process of adding a grade with each successive year of enrollment and did not yet serve all 
high school grades. In addition, two Early Colleges served grades 9-13, or had five-year 
programs. 

On average, 49 percent of the students in the study’s Early Colleges were minority, with a 
range of 12 to 100 percent minority. 12 Three schools served a student population that was at 
least 80 percent Hispanic or African American. On average, 44 percent of the students in 
the Early Colleges were low income (i.e., eligible for the free and reduced-price lunch 
program, FRPL) with a range of 9 to 99 percent. 

Based on our earlier descriptive research, we are able to characterize the implementation of 
the ECHSI in these 10 sites during the years the students attended them (see AIR & SRI, 


10 Calculation of no-show and crossover rates was based on non-missing data. We did not have enrollment records for the first 
year of high school for 9.6 percent of the students in our study sample. 

11 When 2007-08 data were not available, we used 2008-09 data or data for the closest year we could find. 

12 These statistics are for the entire school populations. In subsequent sections we examine the characteristics of our study 
participants. 
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2009, for details on the 2007-08 school survey data). Most of the schools had been open as 
Early Colleges for at least two years as of 2007-08: three opened in 2004-05, three opened in 
2005-06, three opened in 2006-07, and only one was in its first year of implementation in 
2007-08. Six Early Colleges were district-run schools, and the remaining four were charter 
schools. Most of the schools also had a focus in addition to providing opportunities to earn 
college credit: five had a STEM focus, and two had a teacher preparation focus. The Early 
Colleges offered a wide array of supports, with all 10 Early Colleges providing tutoring and 
college preparatory and access information that highlighted scholarships and other financial 
aid information. In addition, Early Colleges offered these supports: 

■ Five Early Colleges offered advisories during the school day. 

■ Seven Early Colleges offered classes in the summer, evenings, or weekends/school 
breaks. 

■ Two Early Colleges had extended school days. 

■ Seven Early Colleges had block scheduling. 

In terms of the implementation of the “college” part of “Early College”: 

■ Seven Early Colleges had course sequences in place to allow students to earn at 
least two years of college credit; 

■ Two Early Colleges had course sequences to allow students to earn at least one 
year of college credit; and 

■ One Early College had a course sequence that allowed students to earn at least 
some college credit. 

Eight of the 10 Early Colleges were located on college campuses. Seven of the 10 Early 
Colleges had a two-year public college partner, two had a four-year public college partner, 
and one had both a two-year and a four-year public college partner. Seven Early Colleges 
had students start college courses in ninth grade, and five Early Colleges noted that all ninth 
grade students took college courses. All but one Early College had college instructors, 
rather than qualified high school instructors, teaching college courses. Only one Early 
College offered Advanced Placement (AP) classes. 

Schools Attended by the Comparison Students 

The comparison students in the study attended 272 different high schools. For enrollment 
and demographic data, we examined the high schools attended by 80 percent of the 
comparison students in the study sample (a total of 62 schools). These schools differed in 
various respects from the 10 Early Colleges in the study. These schools were generally 
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much larger than the Early Colleges, with enrollment ranging from 145 to 2,875 students 
(1,248 students on average). 13 Only 14 of the 62 schools met the typical definition of a small 
school—enrollment of fewer than 150 students per grade. Of the students enrolled in these 
62 schools, 61 percent were minority, and 60 percent were low income (or eligible for FRPL); 
both percentages were higher than those for the Early Colleges in the study. 

To obtain data on the characteristics of the school curriculum, we examined the websites for 
the three high schools most commonly attended by the comparison students of each study 
site (30 schools total). 14 In these schools, AP courses seemed to be more prevalent than dual 
enrollment as a strategy for students to earn college credit. Of the 30 schools attended by 
comparison students, 27 offered AP courses and 14 offered dual enrollment courses. In terms 
of support systems, all 30 schools had guidance and career services available, and some also 
provided specialized supports such as adult mentors for each student, summer programs on 
college campuses, or college prep programs in science and mathematics. 

Overall, the schools attended by Early College students and comparison students differed in 
several noteworthy ways. The Early Colleges in the study were small schools typically located 
on college campuses. The Early Colleges provided students with the opportunity to earn a 
large number of college credits, and they offered an academic support system tailored to 
preparing students for success in college. The majority of the students who did not attend 
Early Colleges enrolled in larger high schools with larger minority and low-income student 
populations. Those schools provided fewer academic supports (e.g., tutoring) and a less direct 
focus on college readiness for all students. 

Data Sources 

To address the study’s research questions, we collected data from multiple sources. We 
gathered educational record data came from Early Colleges, districts, state departments of 
education, and the NSC. For data not available from administrative records, we surveyed 
students after their expected high school graduation date. Each data collection activity is 
described below. Appendix A maps all variables analyzed to the data sources, and 
Appendix B includes more details about data collection. 

Administrative (Educational Record) Data 

We relied on administrative data to measure high school outcomes, student characteristics, 
and postsecondary outcomes. Administrative (or educational record) data have the benefit of 

13 The range and mean excludes an online high school program that served over 8,000 students. 

14 One school attended by the comparison students was an Early College managed by the same intermediary as an Early 
College in the study. It opened as a replication of the study site. Appendix B includes more details about how students 
attending this school were treated in analyses. 
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being authoritative (since they are the official student records) and are the primary source for 
many of the education outcomes and student characteristics examined in this study. We 
collected education record data from multiple sources. Appendix B provides more details on 
these data sources. 

Student Characteristics and High School Outcomes 

We used administrative data to measure student characteristics and high school outcomes. 
When study participants attended and remained enrolled in Early Colleges or in one of the 
school districts or states that provided data, we generally had complete data on their 
background characteristics (including eighth grade achievement) and high school outcomes. 
The source of the data differed by site, and for some sites, we drew from multiple data 
sources. 

Postsecondary Outcomes 

To measure student postsecondary attendance and degree attainment, we relied on data 
from the NSC. 15 We linked the administrative data from schools, districts, and states to the 
NSC postsecondary data. For study students who attended colleges that contribute to the 
NSC and allowed their records to be made available, we received data on their college 
enrollments and degrees earned both during and after high school. 

Student Survey Data 

Because administrative data were not available for some variables of interest (e.g., student 
high school experiences), we also administered a student survey. The student survey 
provided data about the high school experiences for both Early College and comparison 
students, as well as data on high school and longer term outcomes that were not captured 
by administrative data. The survey also included an open-ended item that allowed students 
to comment on their high school and college experiences. 

The survey was drawn from the administrative data sample and included randomly selected 
students who participated in the 2006-07 and 2007-08 lotteries in eight Early Colleges that 
agreed to provide us with student contact information. 16 The two excluded Early Colleges 
could not provide student contact information. We limited the survey sample to students 
from Cohorts 2 and 3. Cohort 1 was excluded because pilot work demonstrated that older 


15 NSC data provide conservative estimates of college enrollment and degree attainment. If a student did not have a record in 
the NSC, then that student was coded as not being in college. However, students may be missing from the NSC because (a) 
they attend a college that does not provide data to NSC, (b) they did not allow NSC to share their individual record data, or (c) 
their name or birthday in our record did not match that in the NSC. We have no reason to expect NSC data to be differentially 
missing for treatment and comparison students. 

16 Student-level non-response-adjusted survey weights were applied to analyses of survey-based outcomes; thus the analysis 
results are generalizable to all participants in the lotteries in the survey sample. 
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students were harder to locate and had difficulty remembering details of their high school 
experiences. 

Across the eight study sites and two cohorts, 1,416 students from the study were sampled 
for the student survey and 1,294 responded. The survey was administered October 2011 to 
January 2012 via web and telephone. The survey response rate was 94 percent for Early 
College students and 88 percent for comparison students, with an overall response rate of 
91 percent. Across sites, response rates ranged from 83 percent to 100 percent. Exhibit 2.4 
shows the number of survey participants in the treatment and comparison groups in each 
cohort. Appendix B provides details about the survey sample, pilot testing and 
administration, response rates, and open-ended data. 


Exhibit 2.4. Number of Early College and Comparison Students in the Survey Sample and Response 
Rates, by Cohort Year 


Cohort Year 

Number of Early 
Colleges in Survey 
Sample 

Number of Early College 
Students Surveyed 
(Response Rate) 

Number of Comparison 
Students Surveyed 
(Response Rate) 

2006-07 

6 

355 

310 



(95%) 

(87%) 

2007-08 

6 

415 

336 



(93%) 

(90%) 

TOTAL 

8 

770 

646 



(94%) 

(88%) 


NOTES: Four of the eight sites have both cohorts represented in the survey sample. 

Data Collection for Each Cohort 

Data were collected during the same timeframe for all cohorts. As a result, the data 
collection timeframe covered different years for each cohort. The oldest cohort had two more 
years of data than the youngest cohort. Exhibit 2.5 displays which type of data were 
collected for each cohort both by academic year and by years after starting ninth grade. 

Note that two Early Colleges had optional five-year programs. They were excluded from 
Cohort 3 because some students in these cohorts would not have had a chance to graduate 
within the study timeframe. 17 


17 In addition, one Early College in our sample had a five-year option for students in Cohort 3. However, almost every study 
student attending this Early College graduated in four years. 
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Exhibit 2.5. Data Collection Activities, by Year and Study Cohort 



/VOTES:The typical educational progression is shown for reference. Students may spend more or less time in high school. 
Cohort 1 did not participate in the student survey. 
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Baseline Equivalence 

The results of our impact study rely on the assumption of random assignment during the 
lottery process. Because we were not in control of the lottery processes, we relied on school 
accounts of their lottery procedures to determine whether students were randomly selected 
to attend the Early College. To verify that students who were selected to attend the Early 
College did not systematically differ from comparison students, we examined the baseline 
equivalence of the two study groups for a variety of student background characteristics, 
including gender, minority status, eligibility for free or reduced-price lunch, first-generation 
status, and ELA and mathematics test scores prior to high school. 18 

We examined group equivalence in two ways. We first conducted a likelihood ratio test of 
global equivalence using a model that predicted treatment assignment using the full set of 
student background characteristics. Appendix B includes a table showing baseline 
equivalence using the global model. We then examined group equivalence in each 
individual characteristic using a model that predicted the characteristic using treatment 
assignment (i.e., lottery winners or not). 19 The global equivalence test shows that there was 
a significant overall difference between the two study groups in the set of student 
characteristics examined (p < .01). We also conducted a global equivalence test excluding 
the prior ELA assessment from the model, and the overall difference between the two study 
groups was not significant. The tests of group equivalence in individual student 
characteristics show that the two groups were not significantly different in any characteristic 
except in prior ELA scores (see Exhibit 2.6). We believe the single significant difference was 
likely due to chance. 20 


18 We also collected student-level data about English language learner (ELL) status and Individualized Education Plan (IEP) 
status before entering high school. However, only a small proportion of students in our sample were ELLs (less than 1 percent) 
or had lEPs (7 percent); therefore we did not include these two variables in the baseline equivalence analyses or as covariates 
in the impact analyses. 

19 All baseline equivalence tests were conducted using two-level models that were similar to the main impact model (see 
Appendix D), which takes into account the clustering of students within lotteries with a random intercept and fixed slopes at the 
lottery level. 

20 We examined prior ELA and mathematics scores in each of the 17 lotteries. We examined cases where the difference 
between the groups was larger than 0.2 standard deviations. For prior ELA scores, Early College students scored higher than 
comparison students in seven lotteries, and comparison students scored higher in two lotteries, leaving eight lotteries with a 
gap of less than 0.2 standard deviations. For prior mathematics scores, Early College students scored higher in four lotteries, 
and comparison students scored higher in four lotteries, leaving nine lotteries with a gap of less than .2 standard deviations. 
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Exhibit 2.6. Background Characteristics of Early College and Comparison Students and Group 
Differences 


Characteristics 

Early College 

Comparison 

Difference 

Female 

51.8% 

55.0% 

-3.2% 

Minority 3 

52.4% 

53.6% 

-1.2% 

First-generation college going 

30.7% 

34.4% 

-3.7% 

Low income 

46.5% 

42.3% 

4.2% 

Prior achievement in ELA 

.25 

.15 

^ «| ** 

Prior achievement in math 

.23 

.29 

-.07 


n = 2,458 students (1,044 Early College, 1,414 comparison) 

NOTES: All data are from administrative data sources except for first-generation status, which comes from the student survey. 
Early College group means are unadjusted means, and comparison group means were computed by subtracting the estimated 
group difference from the unadjusted Early College group means. Prior achievement scores were converted to z-scores and were 
standardized using publicly available statewide assessment means and standard deviations. The values in the Difference column 
may not match the difference between the Early College and Comparison group means due to rounding. 
a Among students who were minority, 73.3 percent were Black, 20.2 percent were Hispanic, 2.9 percent were Asian or Pacific 
Islander, 2.9 percent were multi-racial, and .8 percent were another ethnic or racial group. 

* p < .05; ** p < .01; *** p < .001 


Analytic Methods 

This study was designed to assess the impact of the Early Colleges on a variety of student 
outcomes during high school and beyond. To address our first research question, we 
estimated the impact of Early College on students who were offered admission to an Early 
College based on a lottery. To address our second research question, we examined the 
degree to which the impact of Early Colleges differed for students with different background 
characteristics. In all models used to address these research questions, we included the 
following baseline covariates: female, minority, first generation, low income, prior 
achievement in ELA, and prior achievement in mathematics. The analysis details are 
described in this section. In addition, we describe our approach to handling missing data 
and the generalizability of the study findings. 

Estimating the Impact of Being Offered Admission to an Early College 
(Research Question 1) 

We used two different approaches to estimate the impact of Early College. The intent-to- 
treat (ITT) analysis estimates the impact of being offered admission to an Early College 
through a lottery, regardless of whether the student actually enrolled in the Early College. 
The complier analysis estimates the impact of attending an Early College for students who 
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attended due to winning the lottery. These two types of impact estimates would be 
equivalent if all students complied with their treatment assignment—that is, if all students 
who won the lottery attended the Early College and no students who lost the lottery attended 
an Early College. The estimates may differ, however, if some lottery winners did not attend 
the Early College (i.e., no-shows) or if some who did not win the lottery enrolled in the Early 
College (i.e., crossovers). 

Our primary impact analyses were ITT analyses, in which we compared the outcomes of 
students offered admission to Early Colleges through a lottery (i.e., the treatment group) and 
students not offered admission (i.e., the comparison group). (The model is described in 
further detail in Appendix D.) Defining students’ treatment status on the basis of lottery 
results preserves the benefits of random assignment, maintaining the equivalence of the two 
study groups and thus ensuring the causal validity of the ITT impact estimates. We also 
examined the complier effects of Early Colleges on key outcomes, taking into account both 
no-shows and crossovers (see Appendix G for details about complier analyses). 

To take into account the clustering of students within lotteries, our impact analyses were 
based on a two-level model, where the intercept was modeled as a random effect to 
represent potential differences in average student outcomes across lotteries. For example, 
some lotteries may have attracted more motivated or better prepared students than others. 

We estimated a common treatment effect across all lotteries (based on a fixed-effects 
approach) because the number of lotteries was too small to provide stable estimates of the 
variation in treatment effects across lotteries for some outcomes. The use of a common 
treatment effect means that the results should be generalized only to the set of Early 
Colleges in the sample. In Appendix F, we provide a supplementary analysis of the variation 
in treatment effect for two key outcomes for which a random-effects analysis could be 
conducted without estimation problems. The analysis reported in Appendix F provides 
similar overall results, and it describes the extent to which the impact varies across Early 
Colleges. 

Although we assessed the impact of Early Colleges on a variety of high school and 
postsecondary outcomes, we consider three outcomes “primary”: high school graduation, 
ever attending any type of postsecondary institution, and ever obtaining any type of 
postsecondary degree. 21 We consider the remaining outcomes secondary. 


21 Each of the three primary measures pertains to a unique type of student outcome. Thus, we did not apply multiple 
comparisons corrections to the primary outcomes, because multiple comparisons corrections are typically applied to multiple- 
measures within the same outcome domain. We also did not apply muitiple-comparisons corrections to secondary outcomes. 
We provide the exact p-values of findings for all student outcomes in Appendix E, allowing readers to make corrections 
appropriate for their own purposes. See the What Works Clearinghouse (2011) handbook for a discussion of multiple 
correction standards. 
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Analyses of Variation in Early College Impact Across Students With 
Different Background Characteristics (Research Question 2) 

This study’s second research question asks whether the effect of being admitted to an Early 
College varies across students with different background characteristics. As noted in the first 
chapter, research has consistently found disparities in college degree attainment: minority, 
low-income, and first-generation college-going students are less likely on average to earn 
college degrees than their peers (NCES, 2012b). In fact, one of the guiding principles of the 
ECHSI is to particularly serve these disadvantaged groups of students. 

We examined the potential differential effects of Early College on students with different 
background characteristics by incorporating treatment-by-student-characteristic interaction 
terms into the impact models. We explored whether the effects of being admitted to an Early 
College on key outcomes differed significantly by gender, race/ethnicity, first-generation 
college-going status, low-income status, or level of prior mathematics and ELA achievement. 

For example, we measured the difference in Early College impact on high school graduation 
between minority and non-minority students by adding a treatment-by-minority interaction 
term to the student level of the main impact model. We performed similar analyses to test 
the potential differential effects of Early Colleges associated with gender, low-income status, 
first-generation college-going status, and prior ELA and mathematics achievement scores 
(see Appendix D for details about testing for differences by student demographics). These 
analyses were conducted for three key outcomes: high school graduation, college 
enrollment, and college degree attainment. 

Missing Data 

As explained earlier, data for this study came from the NSC, a student survey, and 
administrative records. We requested postsecondary enrollment and degree attainment data 
from the NSC for each student in our sample. We assumed that students for whom the NSC 
could not find matching records did not attend college or attain a postsecondary degree. 
Therefore, by definition, outcomes that exclusively use NSC data have no missing data. 
Measures based on the student survey had minimal missing data (most were 2 percent or 
less). Measures based on administrative records had higher rates of missing data (for 
example, high school mathematics assessments had a missing rate of 34 percent). 

Appendix C provides detailed information about missing data rates for measures of student 
outcomes and background characteristics. 

To address the potential selection bias caused by non-random missing data, we used a 
multiple imputation approach to impute missing data. (See Appendix C for details.) We 
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generated 10 imputed data sets, conducted all analyses using each imputed data set 
separately, and then combined estimates across the 10 data sets, taking into account the 
uncertainty in imputed values both within and across the imputed data sets. Our primary 
impact analyses were conducted using the full sample with both imputed outcomes and 
imputed covariates. As sensitivity analyses, we also conducted the impact analyses using 
the imputed data sets excluding students with imputed outcomes. Results from these 
sensitivity analyses are presented in Appendix F. These results rarely differed from those 
presented in the text. 

Generalizability of Findings 

Our goal for the study is to assess the degree to which Early Colleges improve student 
outcomes. Such a question could theoretically be answered by randomly assigning students 
to Early College and comparison conditions at a randomly selected set of Early Colleges 
across the country. However, although the randomization of lottery-based admissions offers 
us the opportunity to draw causal conclusions about the impact of Early College, not all 
Early Colleges use a lottery to determine admissions, and the decision about whether or not 
to use an admissions lottery is itself not random. To offer a lottery, a school must have had 
more applicants than it had seats available and must further have decided to make 
admission a random process. Among Early Colleges that used an admissions lottery, the 
study pool was further narrowed to those that had the ability to provide data (particularly 
lottery records) for this retrospective study. Therefore, the final study sample includes a 
nonrandom set of Early Colleges. 

Most of the Early Colleges in our study used multiple criteria to admit students, and the 
admissions lottery applied to only a subset of applicants. For example, many schools 
automatically accepted siblings of current students and children of staff. The study speaks 
only to the impact on applicants within the study schools whose initial offer of admission 
relied on an admissions lottery. 

Due to the purposeful sample of schools and the fact that only some applicants to these 
schools were chosen for admission by lottery, the study findings cannot be generalized to 
schools and students outside of the study sample. However, our focus on Early Colleges for 
which we could verify a random lottery process allows us the unique opportunity to compare 
the outcomes of students who were randomly selected to attend an Early College with the 
outcomes of students who did not win the lottery but would have otherwise attended the 
same Early College. By taking advantage of this naturally occurring experiment, we were 
able to observe the counterfactual, or the outcome that lottery winners would have 
experienced had they not been randomly selected to attend the Early College. Despite its 
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limited external validity, this study enables us to draw causal conclusions about the impact 
of Early College through a rigorous design with strong internal validity. 

Reporting Findings 

In the chapters that follow, we report the estimated impact of Early Colleges on a variety of 
student outcomes. The Early College group means are unadjusted means. Comparison 
group means were computed by subtracting estimated treatment effects from the 
unadjusted Early College group means. Thus, the comparison group means represent the 
outcomes that would have been expected had the Early College students been assigned to 
the comparison condition. Appendix E provides more detailed findings, including estimated 
effect sizes, from all impact analyses. We also provide effect sizes in the report when they 
provide meaningful context for interpreting the findings. 
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Chapter 3. Early College Impact on Student 
Outcomes 

This chapter presents the impact of being admitted to an Early College on student 
outcomes. We first report the findings for high school outcomes, including academic 
achievement and high school graduation. We then report the findings for college outcomes, 
including enrollment, degree attainment, and performance. In addition to the overall Early 
College impact on these outcomes, we also examine whether the Early College impact on 
three key outcomes (high school graduation, college enrollment, and degree attainment) 
differed for students with different background characteristics. 

High School Outcomes 

Achievement 

We measured high school achievement using standardized assessment scores in English 
language arts (ELA) and mathematics, usually measured in 10th grade, and GPA. Our 
previous research suggests that students attending Early Colleges were more likely to be 
proficient on their state assessments than students attending other area high schools, and 
had GPAs of 3.0, about the national average (AIR & SRI, 2009; NCES, 2011). 

In this study, we found that being admitted to an Early College had a statistically significant 
positive impact on student ELA achievement (standardized scores were 0.37 for Early 
College students and 0.23 for comparison students; see Exhibit 3.1 ). 22 If the standardized 
scores were translated into percentiles, Early College students would score at the 64th 
percentile, compared with the 59th percentile for comparison students. However, being 
admitted to an Early College did not have a significant impact on mathematics achievement 
or on GPA. In mathematics, Early College students had an average standardized test score 
of 0.28 and comparison students had an average of 0.23 (see Exhibit 3.1). Although a 
slightly different metric, this finding differs from the findings from research in Texas, which 
indicated that Early College students in Texas had a higher probability of passing high 
school mathematics assessments than did comparison students (SRI, 2011). For both 
subjects, Early College and comparison students were above average for their states. Both 
the Early College and comparison students reported earning a GPA of 2.98, or a “B” 
average (see Exhibit 3.2). Thus, across our three achievement measures, both Early 
College and comparison students were doing well academically in high school. 


22 Mathematics and ELA assessment scores were standardized to have a mean of 0 and standard deviation of 1 based on the 
grade-level mean and standard deviation for each state in a given year. An average score of 0.37, for instance, can be 
interpreted as 0.37 standard deviations above the state average. 
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Exhibit 3.1. High School ELA and Mathematics Assessment Scores, by Study Group 



■ Early College 

■ Comparison 


n = 2,458 students (1,044 Early College, 1,414 comparison) 

SOURCE: Administrative data from 2005-2011 

NOTES: Early College group means are unadjusted means. Comparison group means were computed by subtracting 
estimated treatment effects from the unadjusted Early College group means. 

* p < .05; ** p < .01; *** p < .001 


Exhibit 3.2. High School Grade Point Average, by Study Group 



n = 1,294 students (724 Early College, 570 comparison) 

SOURCE: Student survey data from 2010-11 

NOTES: Early College group means are unadjusted means, and comparison group means were computed by subtracting 
estimated treatment effects from the unadjusted Early College group means. 

* p < .05; ** p < .01; *** p < .001 
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High School Graduation 

Being admitted to an Early College had a statistically significant positive effect on the 
likelihood of graduating from high school (86.0 percent for Early College students and 80.6 
percent for comparison students; see Exhibit 3.3). 23 In fact, the graduation rates of both 
groups compared favorably with the national on-time graduation rate of 75.5 percent 
reported for the 2008-09 academic year, the year in which our study’s oldest cohort was 
expected to graduate (NCES, 2012d). 24 Although the majority of students in both study 
groups graduated from high school, not all students finished at the school where they 
began, and not all students finished. However, there were no significant differences between 
Early College students and comparison students in transfer or dropout rates. 25 See the 
Leaving Early Colleges for Area Alternatives? text box for more information about transfer 
rates for the two groups. 


Exhibit 3.3. Percentage of Students Graduating From High School, by Study Group 



n = 2,458 students (1,044 Early College, 1,414 comparison) 

SOURCE: Administrative data from 2005-2011 

NOTES: Early College group means are unadjusted means, and comparison group means were computed by subtracting 
estimated treatment effects from the unadjusted Early College group means. 

* p < .05; ** p < .01; *** p < .001 


23 These estimates may underestimate eventual graduation rates, as the youngest cohort (Cohort 3) had only four years to 
graduate by the end of our data collection. In the older cohorts, approximately 3 percent of Early College and comparison 
students took five years to graduate (excluding students attending five-year programs). 

24 Note that because two of the Early Colleges in our study are five-year programs, we report an overall graduation rate rather 
than the on-time four-year graduation rate reported by NCES. 

25 We examined student withdrawals (i.e., leaving a high school for any reason), including transferring to another public or 
private high school, beginning home schooling, being expelled, being incarcerated, or dropping out. Both Early College and 
comparison students left the high school they first enrolled in at similar rates (32 percent for Early College students and 37 
percent for comparison students—not a significant difference). Drop out rates, when separated from the overall withdrawal 
rates, are also similar for both Early College and comparison students (about 5 percent for both groups). 
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In addition to the overall impact of Early Colleges on high school graduation, we examined 
whether Early Colleges had differential effects for students with different background 
characteristics (Research Question 2). We found no significant differences in the impact of 
Early College on high school graduation by gender, minority status, first-generation college¬ 
going status, or low-income status. Furthermore, the Early College effects on high school 
graduation were similar for students with different prior ELA and mathematics achievement 
scores. See Appendix E for the detailed results of these subgroup analyses. 


Leaving Early Colleges for Area Alternatives? 


Our earlier research suggested that Early College students may leave their high school because Early Colleges 
tend to be small and students may want the extracurricular activities or elective coursework that come with a 
more traditional high school experience; alternatively, they may not feel ready for the rigors of college 
coursework (AIR & SRI, 2008). Thus, we examined whether Early College students transferred locally to another 
high school more frequently than comparison students.* The percentage of students who transferred to local 
high school alternatives did not differ significantly between the Early College and comparison students: 17 
percent of Early College students and 19 percent of comparison students transferred before graduating or 
dropping out. 

* We defined a local transfer as one where students left the high school to attend a different high school in the district, to attend a 
private school, or to be home schooled. 


College Outcomes 

Information about students’ academic performance and graduation rates sheds light on the 
effectiveness of Early Colleges in producing high school success. Flowever, the ultimate 
goal of the ECHSI is for students to earn a postsecondary degree or credential that prepares 
them for successful entry into the workforce. This section examines the effectiveness of 
Early Colleges in improving key college outcomes for students: college enrollment, degree 
attainment, and performance in college. 

College Enrollment 

Because Early Colleges build in college course-taking as part of their design, we should 
expect Early College students to enroll in college courses at higher rates during their high 
school years than comparison students (the next chapter on high school experiences 
examines college enrollment during high school in detail). Flowever, Early College and 
comparison students alike had the opportunity to enroll in college following high school 
graduation. Therefore, in our impact analyses, we examined whether students in the study 
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ever enrolled in college. 26 We found that being admitted to an Early College had a 
statistically significant positive impact on college enrollment: 79.7 percent of Early College 
students had at least one record of college enrollment, roughly 9 percentage points higher 
than the college enrollment rate for comparison students, 70.5 percent (see Exhibit 3.4). 
Our analyses of differential Early College impact indicated that the Early College impact on 
college enrollment was similar for students with different background characteristics (i.e., 
gender, minority status, first-generation college-going status, low-income status, or prior 
achievement). See Appendix E for detailed results from the differential impact analyses. 


Exhibit 3.4. Percentage of Students Who Ever Enrolled in College and Who Enrolled by Year 4 and by 
Year 5 After Starting Ninth Grade, by Study Group 


100 % 



Ever By Year 4 By Year 5 

Time Period 


■ Early College 
□ Comparison 


n = 2,458 students (1,044 Early College, 1,414 comparison) 

SOURCE: National Student Clearinghouse, 2005-2012 

NOTES: Early College means are unadjusted means, and comparison group means were computed by subtracting estimated 
treatment effects from the unadjusted Early College means. 

* p < .05; ** p < .01; *** p < .001 


To cast further light on the impact of Early Colleges on college attendance, we conducted 
additional analyses examining college enrollment by students’ fourth and fifth years after 
starting ninth grade. The enrollment rates by Year 4 reflect the time period when students 
would traditionally still be in high school. By this point in time, Early Colleges had a 
significant impact on rates of college enrollment: 63.1 percent of Early College students had 
at least one record of college enrollment during high school, compared with 22.9 percent of 
comparison students (see Exhibit 3.4). 


26 “Ever” indicates “at any time during our study period.” Therefore, students in the oldest cohort have had a longer period of 
time to enroll in and graduate from college compared to students in the youngest cohort. In our models, we account for this 
difference, as students are only compared within each cohort. 
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We also examined enrollment rates by the end of Year 5. This measure is important for two 
reasons. First, it represents a timeframe when all students on a traditional trajectory would 
have had an opportunity to finish high school and to enroll in college after high school 
completion. 27 Second, it represents the latest point in time for which we have data for all 
study cohorts. By the end of Year 5, 77.4 percent of Early College students and 66.7 
percent of comparison students had enrolled in college (see Exhibit 3.4). Between Year 4 
and Year 5, the gap between the groups narrowed from 40.2 to 10.7 percentage points. 28 
Despite this decrease in the gap, the difference between the two groups remained 
statistically significant. Although Year 5 (one year past high school) is the last point at which 
the entire study sample had data available, we had data for two years beyond high school 
(i.e., Year 6) for the two oldest cohorts. The College Enrollment Trends for the Two Oldest 
Cohorts text box explores enrollment patterns for these two cohorts. 


Note that this is not universally true. As described in Chapter 2, some students in the study—primarily those attending Early 
Colleges—attended a school with an optional fifth year. 

28 This finding suggests that 14.3 percent of Early College students (the difference between the percentages in Year 4 and 
Year 5) enrolled in college for the first time in Year 5, and that 43.8 of comparison students did the same. 
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College Enrollment Trends for the Two Oldest Cohorts 


Two student cohorts have data available for six years after entering ninth grade. By examining student 
outcomes for two years past typical high school graduation, we are able to get a sense of what we 
might observe if we were to track the full study sample for additional years. 

College enrollment rates increased for both Early College and comparison students between Year 4 and 
Year 6, with comparison students closing a sizeable portion of the gap in Year 5—the period of time 
during which students on a traditional trajectory would have completed high school and had the 
opportunity to move on to college. While the gap in enrollment rates between the two groups declined 
from 39.2 percentage points in Year 4 to 10.3 percentage points in Year 5, it remained stable at 9.8 
percentage points by the end of Year 6. Differences in all three years were statistically significant. These 
findings suggest that while the gap between the groups narrowed after the conclusion of high school, 
comparison students did not catch up to their Early College counterparts as of two years after high school. 


Percentages of Students Who Enrolled in College by Year 4, by Year 5, and by Year 6 After Starting Ninth 
Grade for the 2005-06 and 2006-07 Cohorts, by Study Group 
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Early College 
Comparison 


n = 1,458 students (564 Early College, 894 comparison) 

SOURCE: National Student Clearinghouse, 2005-2012 

NOTES: Early College means are unadjusted means, and comparison means were computed by subtracting estimated 
treatment effects from the unadjusted Early College means. 

* pc.05;** pc.01;*** pc.001 


In addition to examining whether and when students enrolled in college, we also examined 
where they enrolled. Our descriptive studies indicate that Early Colleges primarily partner 
with two-year institutions of higher education: 76 percent of Early Colleges in the ECHSI in 
2007-08 partnered with two-year colleges, while 35 percent partnered with four-year 
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colleges; 11 percent partnered with both (AIR & SRI, 2009). As described in Chapter 2, our 
study sites have a similar composition: 8 of 10 Early Colleges partnered with two-year 
colleges, 2 partnered with four-year colleges, and 1 partnered with both. 29 The fact that the 
majority of Early Colleges partnered with two-year colleges has led to the criticism that Early 
Colleges may funnel students primarily into two-year colleges and away from educational 
pathways that would lead to bachelor’s degrees. 

Our results indicate that being admitted to an Early College had a statistically significant 
positive impact on attending both two-year and four-year colleges. The percentage of Early 
College students who attended two-year colleges (59.2 percent) was significantly higher 
than the percentage for comparison students (37.8 percent) (see Exhibit 3.5). This 
difference might be expected, particularly during the high school years, when Early Colleges 
frequently partner with local two-year institutions. Indeed, a gap of more than 36 percentage 
points existed between the two groups by the end of Year 4 (48.0 percent for Early College 
students and 11.6 percent for comparison students), which would mark the end of the high 
school experience for students on a traditional trajectory. By the end of Year 5, the gap 
shrank to 25 percentage points (55.7 percent for Early College students and 31.1 percent for 
comparison students), but it remained statistically significant. 


Exhibit 3.5. Percentage of Students Who Ever Enrolled in a Two-Year College and Who Enrolled in a 
Two-Year College by Year 4 and by Year 5 After Starting Ninth Grade, by Study Group 



■ Early College 

■ Comparison 


n = 2,458 students (1,044 Early College, 1,414 comparison) 

SOURCE: National Student Clearinghouse, 2005-2012 

NOTES: Early College means are unadjusted means, and comparison group means were computed by subtracting estimated 
treatment effects from the unadjusted Early College means. 

* p < .05; ** p < .01; *** p < .001 


29 To place these figures in the national context, 29 percent of full-time college enrollments in fall 2010 were at two-year 
institutions and 71 percent were at four-year institutions; 65 percent of part-time fall college enrollments in fall 2010 were at 
two-year institutions and 35 percent were at four-year institutions (NCES, 2012f). 
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Perhaps more notable is the trend for enrollment at four-year colleges. If Early Colleges 
funnel students primarily towards attending two-year institutions, we might see lower rates of 
four-year college attendance for Early College students than for comparison students. In 
fact, the opposite is true. Being admitted to an Early College had a statistically significant 
positive impact on ever attending a four-year college (see Exhibit 3.6). 30 Although the gap in 
four-year college enrollment between Early College students and comparison students 
narrowed from Year 4 to Year 5, it remained statistically significant. By Year 5, 48.9 percent 
of Early College students and 43.2 percent of comparison students had enrolled in a four- 
year college. 


Exhibit 3.6. Percentage of Students Who Ever Enrolled in a Four-Year College and Who Enrolled in a 
Four-Year College by Year 4 and by Year 5 After Starting Ninth Grade, by Study Group 



Time Period 


■ Early College 
oi Comparison 


n = 2,458 students (1,044 Early College, 1,414 comparison) 

SOURCE: National Student Clearinghouse, 2005-2012 

NOTES: Early College means are unadjusted means, and comparison group means were computed by subtracting estimated 
treatment effects from the unadjusted Early College means. 

* p < .05; ** p < .01; *** p < .001 


The findings for four-year enrollments are particularly important because enrollment in a 
four-year institution is less likely for students from disadvantaged groups, even if 
academically prepared. For example, Florn and Nunez (2000) found that even after 
controlling for academic qualifications, first-generation students were less likely than their 
peers to enroll in a four-year college. The impact study findings indicate that Early Colleges 
were not guiding students away from postsecondary paths that would lead to bachelor’s 
degrees. For additional information about students’ enrollment, see the text box, Do Early 
College Students Attend Early College Partner Institutions? 


30 Students could, of course, attend multiple types of colleges. 
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Do Early College Students Attend Early College Partner Institutions? 


Our descriptive evaluation of Early Colleges found that one of the primary motivating factors for postsecondary 
institutions to partner with Early Colleges was the opportunity to build a pipeline for student enrollment (AIR & 
SRI, 2009). However, the pipeline may already be strong; institutions designed to serve their local 
communities, especially the two-year public colleges that compose the majority of Early College partners, may 
attract students from all local high schools, with or without the Early College intervention. If Early Colleges 
produce higher student enrollments in the partner institution, other colleges and universities may have a 
stronger motivation to develop similar relationships with high schools in the future. Our study found that after 
high school graduation (when attendance at a partner institution would be a student’s choice and not a 
structural feature of their high school), the percentage of Early College students attending the Early College’s 
partner institution was approximately 12 percentage points higher than that for comparison students (55.8 
percent for Early College students and 43.7 percent for comparison students), a statistically significant 
difference.* The findings suggest that from a college perspective, an Early College partnership does in fact 
expand the pipeline of students. 

* This measure also incorporates students who earned a degree at the partner institution, even if they did so while in high 
school. 


Degree Attainment 

The results above describe student exposure and access to college. Enrollment, however, is 
important only as an indicator of students’ likelihood of attaining a postsecondary credential 
or degree. Research and job projections continue to highlight a college degree as the key to 
meaningful participation in the workforce, resulting in increased earning potential and 
opportunity for advancement. Thus, the fundamental question for the study is whether Early 
College students were more likely to earn postsecondary degrees. 

Because Early Colleges start students working toward a college degree, we should expect 
Early College students to complete college degrees at higher rates than comparison 
students during or soon after high school. However, comparison students may catch up in 
later years. Therefore, we also consider the year in which students completed their college 
degree. 

The results indicate that being admitted to an Early College had a statistically significant 
positive impact on degree attainment. By the end of the 2011-12 academic year, 22.2 
percent of Early College students earned postsecondary degrees, compared with 1.9 
percent of comparison students (see Exhibit 3.7). 
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Exhibit 3.7. Percentage of Students Ever Earning a Postsecondary Degree, by Degree Type and Study Group 



n = 2,458 students (1,044 Early College, 1,414 comparison) 

SOURCE: National Student Clearinghouse, 2005-2012 

NOTES: Early College means are unadjusted means, and comparison group means were computed by subtracting estimated 
treatment effects from the unadjusted Early College means. 

* p < .05; ** p < .01; *** p < .001 

The significant positive treatment effect held for associate’s and bachelor’s degrees, but not 
for certificates. Twenty-one percent of Early College students earned an associate’s degree, 
compared with 1.4 percent of comparison students (see Exhibit 3.7). Bachelor’s degree 
attainment understandably occurred at lower rates, as no students in our study had an 
opportunity to attend college for the four years after high school that a bachelor’s degree 
traditionally requires. Nevertheless, a significant difference between the study groups existed, 
with 2.3 percent of Early College students earning a bachelor’s degree and virtually no 
comparison students doing the same. 31 Certificate attainment for the two groups of students 
was similar, with few students in either group earning a certificate (1.2 percent of Early College 
students and 0.7 percent of comparison students). The majority of certificate programs require 
one year of coursework or less to complete, meaning that students in the study should have 
had the opportunity to complete a certificate program if that was their desired path. The low 
rate of certificate attainment for both groups suggests that students in the study who pursued 
a college education did so primarily by enrolling in degree-granting programs. 

In addition to the overall effect of Early Colleges on degree attainment, we also examined 
whether the Early College effect differed by student background characteristics (Research 
Question 2). Although the effect of Early Colleges on degree attainment was similar for first- 

31 Note that the sum of the percentage of students earning associate’s degrees and bachelor’s degrees exceeds the overall 
percentage of students earning any degree because many of the students who earned a bachelor’s degree had already earned 
an associate’s degree. 
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generation students and non-first-generation students, it differed significantly for the student 
subgroups defined by the following characteristics: 

■ Gender: The Early College impact on college degree attainment was significantly 
stronger for females than for males. Female Early College students were 
approximately 19 times more likely to receive a college degree than female 
comparison students (22.8 percent vs. 1.2 percent). 32 Male Early College students 
were approximately 7 times more likely to obtain a college degree than male 
comparison students (21.7 percent vs. 2.9 percent). 

■ Race: The Early College impact on college degree attainment was significantly 
stronger for minority than for white students. Among minority students, Early College 
students were 29 times more likely to obtain a college degree than comparison 
students (26.5 percent vs. 0.9 percent). Among white students, Early College 
students were approximately 8 times more likely to obtain a college degree than 
comparison students (22.8 percent vs. 2.8 percent). 

■ Income: The Early College impact on college degree attainment was significantly 
stronger for low-income students. Specifically, low-income Early College students 
were approximately 25 times more likely than low-income comparison students to 
obtain a college degree (19.7 percent vs. 0.8 percent). In contrast, higher income 
Early College students were approximately 7 times more likely to obtain a degree 
than higher income comparison students (24.5 percent vs. 3.4 percent). 

■ Prior Achievement: The impact of Early College on college degree attainment was 
significantly stronger for students who entered high school with better mathematics 
and ELA scores. The difference between Early College students and comparison 
students was smallest among students with lower prior ELA and mathematics test 
scores and largest among students with higher ELA and mathematics test scores. 

Appendix E provides additional details on these analyses. 

As we did for college enrollment, we moved beyond overall measures of college degree 
attainment to examine degree attainment by Year 4 and Year 5. 33 The data suggest that the 
overwhelming majority of Early College students who earned a college degree did so during 
high school. Of all Early College students, 19.8 percent had received at least one degree by 
Year 4, and 21.4 percent had received at least one degree by Year 5 (see Exhibit 3.8). In 
contrast, very few comparison students had earned a college degree by either point in time 


32 Although the text expresses results in terms of the ratio of the probability of receiving a college degree for Early College and 
comparison students, the analyses are based on the ratio of the odds of attending college. See Appendix E for details. 

33 For students following a traditional trajectory, the fourth and fifth years would represent the final year of high school and the 
first year after high school, respectively. By referring to the number of years after beginning ninth grade, we were able to 
examine our entire sample on the same playing field, including those students who dropped out of high school, those who took 
extra time to earn their high school diploma, and those for whom we lacked high school graduation data. 
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(0.7 percent and 1.2 percent, respectively). See the College Degree Attainment Trends for 
the Two Oldest Cohorts text box for degree attainment patterns through Year 6 for our two 
oldest cohorts. 


Exhibit 3.8. Percentage of Students Who Earned a Postsecondary Degree by Year 4 and by Year 5 After 
Starting Ninth Grade, by Study Group 


100 % 


< 



By Year 4 By Year 5 

Time Period 


■ Early College 

■ Comparison 


n = 2,458 students (1,044 Early College, 1,414 comparison) 

SOURCE: National Student Clearinghouse, 2005-2012 

NOTES: Early College means are unadjusted means, and comparison group means were computed by subtracting estimated 
treatment effects from the unadjusted Early College means. 

* p < .05; ** p < .01; *** p < .001 
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College Degree Attainment Trends for the Two Oldest Cohorts 


We again explore data for our two oldest cohorts, which have data available for six years after beginning 
ninth grade. This sixth year is particularly interesting for tracking college degree attainment, as students 
on a traditional trajectory would have had two years after high school completion to earn an associate’s 
degree. These analyses are also suggestive of what we might see if we could track the full study sample 
for additional years. 

The percentage of Early College students who earned a postsecondary degree increased from 
17.6 percent by the end of Year 4 to 20.0 percent by the end of Year 6. In contrast, the percentage of 
comparison students who completed a postsecondary degree increased from 0.3 percent to 1.7 percent 
over the same time period, for a gap of over 18 percentage points between the two groups. This 
difference remained stable and statistically significant; comparison students did not catch up to their Early 
College counterparts as of two years after high school. 


Percentages of Students Who Attained a College Degree by Year 4, by Year 5, and by Year 6 After 
Starting Ninth Grade for the 2005-06 and 2006-07 Cohorts, by Study Group 



Early College 
Comparison 


n = 1,458 students (564 Early College, 894 comparison) 

SOURCE: National Student Clearinghouse, 2005-2012 

NOTES: Early College means are unadjusted means, and comparison means were computed by subtracting estimated 
treatment effects from the unadjusted Early College means. 

* p < .05; ** p < .01; *** p < .001 
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College After High School 

Our findings for college enrollment and degree attainment suggest that most of the 
difference between the two study groups took place within four years after starting ninth 
grade. Although some of the gaps shrank after high school, especially for college enrollment 
rates, the gaps persisted and remained statistically significant through the end of Year 5; we 
found no evidence that the comparison students caught up to their Early College peers. 
Nevertheless, the findings raise an important question: what impact do Early Colleges have 
after students leave the highly structured and scaffolded high school environment? Because 
we tracked our full study sample only through the end of Year 5, our data do not allow us to 
make inferences about the long-term degree attainment rates that would be most useful for 
answering this question. Nevertheless, we explored post-high school outcomes to better 
understand the impact of Early Colleges. For the purposes of this discussion, “after high 
school” refers to the period of time after the end of Year 4. 34 

We first examined college enrollment after the end of Year 4 and found no statistically 
significant difference between the rates for Early College students (71.1 percent) and 
comparison students (68.4 percent; see Exhibit 3.9). 35 The overall enrollment rates for both 
groups compare favorably with national data, which indicate that 68 percent of high school 
graduates in 2010 enrolled in two-year or four-year colleges immediately following high 
school completion; 36 the percentage drops for low-income students (52 percent), African 
American students (66 percent), and Hispanic students (60 percent) (NCES, 2012e). 

Students in both groups also attended two-year colleges at comparable rates after Year 4, 
with 32.8 percent of Early College students and 34.8 percent for comparison students (see 
Exhibit 3.9). Nationally, 26 percent of high school completers enrolled in two-year colleges 
after high school in 2011 (NCES, 2013). 

The story differs when examining enrollment in four-year colleges. Early Colleges had a 
statistically significant positive impact on enrollment at four-year colleges after high school: 
51.2 percent of Early College students and 45.5 percent of comparison students attended a 
four-year college after high school. Therefore, even though most of the Early Colleges 


34 For students following a traditional trajectory, these post-Year 4 outcomes should correspond to the years after high school. 
Flowever, as noted in Chapter 2, some students participate in five-year high school programs and might still be in high school 
after the conclusion of Year 4. 

35 We use Year 4 here rather than high school graduation to take full advantage of available data; whereas we have Year 4 
outcomes available for all students in our study, students who drop out of high school or lack graduation dates in our data 
would otherwise need to be dropped from our analysis, thus lowering our sample size for these variables. Sensitivity analyses 
that used the high school graduation date instead of Year 4 to define “after high school” produced college enrollment rates that 
were higher for both groups (81.3 percent for Early College students and 80.2 percent for comparison students), but these 
rates remained statistically indistinguishable. 

36 Note that these national data for overall attendance rates and attendance rates at 2-year and 4-year colleges refer only to 
high school completers. In contrast, the percentages reported in this study refer to the entire pool of students who applied to 
Early Colleges. 
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partnered with two-year colleges, Early College students were more likely to attend four-year 
institutions than comparison students, both during and after high school. Both study groups 
had higher rates of attending four-year colleges after high school than the national rate of 42 
percent (NCES, 2013). 


Exhibit 3.9. Percentage of Students Enrolling in College Between Year 5 and Year 7 After Starting Ninth 
Grade, by Study Group 



Type of Institution 


■ Early College 

■ Comparison 


n = 2,458 students (1,044 Early College, 1,414 comparison) 

SOURCE: National Student Clearinghouse, 2005-2012 

NOTES: Early College means are unadjusted means, and comparison group means were computed by subtracting estimated 
treatment effects from the unadjusted Early College means. 

* p < .05; ** p < .01; *** p < .001 


We also examined rates of degree attainment after high school. Overall, Early College 
students had higher rates of degree attainment after Year 4: 4.5 percent of Early College 
students earned a postsecondary degree or credential, as compared with 1.5 percent of 
comparison students (see Exhibit 3.10). 37 This Early College impact held for both 
associate’s degrees and bachelor’s degrees. Two percent of Early College students earned 
an associate’s degree after Year 4, as compared with 0.7 percent of comparison students; 
2.3 percent of Early College students earned a bachelor’s degree, compared with virtually 
no comparison students. The differences for all three measures were statistically significant. 


37 As we did for college enrollment, we use Year 4 rather than high school graduation to take full advantage of available data; 
whereas we have Year 4 outcomes available for all students in our study, students who dropped out of high school or lacked 
graduation dates in our data would otherwise need to be dropped from our analysis, thus lowering our sample size for these 
variables. Analyses that used high school graduation date to define “after high school" produced degree attainment rates that 
were higher for both groups (10.8 percent for Early College students and 3.3 percent for comparison students); the difference 
between the groups was statistically significant. 
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Exhibit 3.10. Percentage of Students Who Earned Postsecondary Degrees Between Year 5 and Year 7, by 
Degree Type and Study Group 



■ Early College 

■ Comparison 


n = 2,458 students (1,044 Early College, 1,414 comparison) 

SOURCE: National Student Clearinghouse, 2005-2012 

NOTES: Early College means are unadjusted means, and comparison group means were computed by subtracting estimated 
treatment effects from the unadjusted Early College means. 

* p < .05; ** p < .01; *** p < .001 


What do these findings tell us about the impact of Early Colleges during and after high 
school? The results from this study suggest that the impact of Early Colleges occurred 
primarily during the high school years—it is in this timeframe that the magnitude of the group 
differences was the largest, and where enrollment and degree attainment results were 
consistently statistically significant. After high school, significant differences continued to 
exist in four-year college enrollment and degree attainment. The reader may be tempted to 
conclude from these findings that Early Colleges provide limited benefit after high school. 
However, the more important interpretation may be that Early Colleges provided students 
with an edge over the comparison students during high school, and the edge persisted even 
after high school. A complete understanding of how long the Early College impact persists 
would require tracking degree attainment and workforce outcomes over a more extended 
time period. 


Although questions about the long-term impact of Early Colleges remain, the available 
evidence strongly indicates that Early College students complete college ahead of their 
comparison group peers. Even if comparison students were to catch up over time, Early 
Colleges offer the benefit of acceleration. Students who earn degrees earlier have the 
opportunity to enter the workforce earlier and potentially realize additional lifetime earnings. 
At the same time, earning a college degree while in high school can save money for 
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students and their families, as Early Colleges often cover most, if not all, of the college costs 
incurred during high school (AIR & SRI, 2008). 

Student survey responses suggest some awareness of these benefits among Early College 
students. 38 One student attested to the cost savings benefits by explaining, “This was 
probably the best decision of my life. It has helped me get some college education that I 
might not have been able to afford otherwise.” Other Early College students highlighted the 
benefits of accelerating degree attainment. According to one, “Early College was great 
preparation for the real world and helped me earn a degree several years earlier than most 
people that I know. At first, it was difficult to be in college at such an early age, but I got 
used to it after a couple years and I am very thankful for the experience.” 

College Performance 

In this section, we explore students’ college performance after high school in terms of 
remediation, grades, perceptions of academic difficulty, and persistence. These analyses 
serve two purposes. First, they provide insight into the extent to which students might be on 
track to earn a postsecondary degree or credential. Second, the data provide an additional 
window into students’ successes after leaving the highly structured and supportive Early 
College environment. The Early College charge is not only to provide access to college 
while students are in high school, but also to prepare them to continue in college after high 
school, particularly if they have not yet earned a degree. 

The following analyses include only those students who attended college after high school. 39 
Unlike students in the overall study sample who were randomly assigned to treatment and 
comparison conditions, this subset of students—students who attended college after high 
school—was not randomly assigned. It is possible that among this subset of students, Early 
College students might be systematically different from the comparison students. 40 
Therefore, the following results should be interpreted with caution. 

• Developmental Coursework: One measure of students’ success in the transition to 
college is whether they are placed in non-credit-earning, developmental courses; 
enrollment in these remedial courses suggests a high school experience that 
insufficiently prepared the student for college academics, and some research has 
suggested that it is also associated with a decreased likelihood of college graduation 


38 At the end of the survey, students were asked, “Would you like to tell us anything else about your high school or college 
experiences?” See Appendix B for more information on the open-ended response data and analysis. 

39 The specific sample for each measure depends on the data source and measure. See the additional footnotes associated 
with each measure for more information. 

40 As discussed earlier in the chapter, roughly 70 percent of both Early College students and comparison students enrolled in 
college after Year 4 according to NSC data. However, the differences in unobserved characteristics—which are addressed 
through the randomization process that initially divided students into Early College and comparison groups—are not accounted 
for here. 
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(e.g., Adelman, 1999; Complete College America, 2012). Data from the student 
survey suggest that overall, approximately one fifth of all students who attended 
college after high school enrolled in a college developmental course in mathematics, 
reading, or writing (18 percent of Early College students and 22 percent of 
comparison students, not a statistically significant difference). 41 

■ College GPA: A second indicator of students’ success in college is their performance 
in college courses. We expected that students who were equipped for success in 
college coursework would earn better grades than underprepared students would. 
The survey asked students who completed at least one term of college after high 
school to report on their college grades. 42 Both Early College and comparison 
students reported about a “B” average, with a GPA of 3.1 for both groups. 

■ Perceptions of Academic Difficulty: Students’ perceptions of their transition to college 
might reveal potential differences in high school preparation between the study 
groups. On the student survey, Early College students reported experiencing 
significantly less difficulty in college than their comparison peers (2.0 versus 2.2 on a 
4-point scale on which 1 means “very easy” and 4 means “very difficult”; see 
Appendix A for details about the scale). 43 The effect size is 0.22. 

■ Persistence in College: By examining the extent to which students continue to enroll 
in college in consecutive years, we can assess whether students are on the path 
toward degree attainment. Of the students who had NSC records of college 
enrollment in Year 5, the level of persistence to the fall of Year 6 was comparable 
between Early College students (67.4 percent) and comparison students (66.4 
percent). 44 

The differences between the groups in terms of developmental coursework, grades, 
perceived difficulty, and persistence were small when they existed at all. Further study might 
help expand our understanding about the ways in which Early Colleges impact not only 
college access but also college performance and degree attainment. 


41 Data on developmental coursework came from the student survey for the subsample of students who reported attending 
college after finishing or leaving high school {n = 1,002; 562 Early College, 440 comparison). 

42 Since the survey was administered in fall 2011, data on college GPA were primarily applicable to the older of the two survey cohorts 
who, on a traditional path, could have completed up to a year of college at the time of the survey. The younger survey cohort had 
typically just graduated from high school and were beginning their first year of college after high school, so they had not yet earned any 
college grades. Thus, this item reflects a reduced sample size of 455 students (253 Early College, 202 comparison) who reported 
completing at least one term of college “after finishing or leaving high school.” 

43 Data on academic difficulty came from the student survey for the subsample of students who had completed at least one term 
of college after finishing or leaving high school, (n = 455; 253 Early College, 202 comparison). 

44 This measure of persistence uses NSC data, and includes students who attended college after high school, where “after high 
school” refers to the time period after the completion of a student’s fourth year after entering ninth grade. 
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Conclusion 

The data we collected indicate that the Early Colleges in our sample were highly effective at 
getting students on the path to a college degree. Some significant impacts existed on high 
school outcomes—most notably high school graduation—but the real story is evident in the 
findings for college degree attainment, which consistently demonstrate a significant Early 
College impact. 

In high school, being admitted to an Early College had a significant positive impact on 
students’ performance on ELA state assessments and on the students’ likelihood of 
graduating from high school. The Early Colleges in our sample did not have a statistically 
significant impact, however, on students’ high school mathematics scores and GPAs. 

We found that that being admitted to an Early College had a significant positive impact on 
students’ college enrollment and degree attainment. Early College students were more likely 
than comparison students to enroll in college overall, and at both two-year and four-year 
colleges. Because Early College students start college in high school, we should expect 
Early College students to complete college degrees earlier. However, Early College 
students were more likely to earn college degrees at every point in time we examined. 

We also examined whether the effects of Early Colleges differed for students with different 
background characteristics. We found that the effect of Early College on graduating from high 
school and college enrollment did not differ for students with different characteristics. 

However, the effect of Early College on college degree attainment was stronger for female, 
minority, and low-income students, as well as students with higher levels of prior achievement 
relative to their peers. 

After leaving high school, Early College students were more likely than comparison students 
to enroll in a four-year institution and to have earned a college degree. These persistent 
differences are likely due to the head start Early College students received while in high 
school. However, Early College students had about the same likelihood as comparison 
students of enrolling at a two-year college after high school, and overall college-going rates 
were also comparable. For the subset of students who enrolled in college after high school, 
Early College and comparison students performed comparably on measures related to 
college performance and persistence when attending college after high school. 

Our analyses suggest that the college enrollment and degree attainment differences between 
Early College and comparison students were largest by the end of the traditional high school 
experience (four years after entering ninth grade). These differences narrowed in later years, 
but statistically significant gaps remained on measures of whether students had ever enrolled 
in college or earned a degree. Because our data sources only capture outcomes for our 
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complete sample through the fifth year after starting ninth grade, we cannot definitively answer 
the question of whether comparison students might eventually catch up to Early College 
students. Further study that considers outcomes for additional years beyond high school is 
required to determine whether long-term degree attainment and workforce outcomes are 
different between Early College and comparison students. Such research might help us 
understand whether Early Colleges act primarily as an accelerating mechanism, or whether 
they also produce higher degree attainment and job earnings for students over time. 
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Chapter 4. High School Experiences 

The previous chapter showed that Early College students fared significantly better than 
comparison students on a variety of high school and college outcomes. As the introduction 
to this report described, Early Colleges aim to provide students with rigorous academics and 
a comprehensive academic and social support system to equip them with the skills 
necessary for college completion. This chapter explores several key features of Early 
Colleges to understand the extent to which the treatment and comparison students differed 
in their experiences while in high school. Specifically, we examined students’ exposure to 
college in high school, the academic rigor of the courses students took in high school, and 
student supports offered by high schools. Most of these measures come from the student 
survey, administered to students in Year 5 or Year 6 after entering high school. Average 
differences on survey scales can be difficult to interpret, so we report effect sizes throughout 
this chapter to help interpret the size of the differences. 45 

College Exposure in High School 

One of the ECHSI Core Principles is for students to earn at least one year of college credit 
in high school. To accomplish this, students must enroll in college and successfully 
complete their college courses. This section examines both Early College and comparison 
students’ patterns of college enrollment and credit accrual in high school. College enrollment 
was measured using NSC data and credit accrual was measured using student survey data. 

College Enrollment in High School 

Chapter 3 demonstrated that Early College had a significant positive impact on college 
enrollment during high school. To understand the pattern of college access throughout high 
school, we examined college enrollment in each high school grade. As Exhibit 4.1 shows, 
Early College had a statistically significant positive impact on college enrollment in each 
grade, although both groups were more likely to enroll in college during the later years of 
high school. In ninth grade, 28.9 percent of Early College students and almost no 
comparison students enrolled in college. In 12th grade, enrollment rates for both groups 
increased, but a gap remained; over half (52.5 percent) of Early College students and about 
one fifth (19.4 percent) of comparison students enrolled in college. 


45 According to the What Works Clearinghouse (2011), effect sizes that are at least 0.25 are generally considered 
“substantively important” in the field of education. A substantively important effect indicates that there is a meaningful 
difference between groups on a given outcome measure. 
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Exhibit 4.1. Percentage of Students Who Enrolled in College in High School, by Grade and Study Group 


100 % 

CD 


<3 80% - 

c= 



9th 10th 11th 12th Ever in high 

school 


High School Grade 


■ Early College 

■ Comparison 


n = 2,458 students (1,044 Early College, 1,414 comparison) 

SOURCE: National Student Clearinghouse, 2005-2012 

NOTES: Early College means are unadjusted means, and comparison group means were computed by subtracting estimated 
Early College effects from the unadjusted Early College means. 

$ Too few students to report 
* p < .05; ** p < .01; *** p < .001 

As noted in the introduction, nationwide in 2010-11, less than 10 percent of high school 
students enrolled in college courses; Early College students’ college enrollment rate 
exceeded this national rate in each year of high school. The rates for comparison students 
were quite high in 12th grade as well. These findings suggest that the study students were 
highly motivated to attend college. 


Although the goal of the Early Colleges is to get all students enrolled in college while in high 
school, our findings suggest that the Early Colleges did not meet this goal. Almost 37 
percent of Early College students did not enroll in college while in high school (see Exhibit 
4.1). There are three possible reasons for this. First, as noted in Chapter 2, the NSC 
provides conservative estimates for college enrollment. Second, some students in our Early 
College group never attended an Early College, and others left the Early College before high 
school graduation. Third, not all students enrolled in college during high school. This last 
explanation is consistent with our previous research; we found that about one third of upper 
grade students were not enrolled in college classes (AIR & SRI, 2008, 2009). Our prior work 
found that the extent to which Early Colleges structure college course-taking for their 
students varies substantially. See the text box, How Early Colleges Integrate College 
Courses, for additional details. 
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How Early Colleges Integrate College Courses 


We know a good deal about how schools in the ECHSI integrate college coursework from our previous research, 
including timing, course sequencing, and course location. The timing of college enrollment varies across Early 
Colleges, though most Early Colleges introduce students to college courses early in high school. For example, 
more than half of Early Colleges in the ECHSI (59 percent) had at least some students enrolled in credit-bearing 
college classes in ninth grade (AIR & SRI, 2009). Early Colleges also vary in how they structure the course 
sequence. Some Early Colleges prescribed the college courses available to students. Early Colleges reported 
that this approach ensures that students not only meet high school graduation requirements but also accumulate 
a meaningful and transferable set of classes for use in future enrollment at four-year colleges (AIR & SRI, 2009). 
The exhibit below includes a sample Early College’s college course sequence, which illustrates a common Early 
College approach of gradually giving students more college-level courses in the later high school grades. 

Integrating College Courses at an Early College Located on a College Campus 


Grade 

College Courses’ Instructors and Students 

College Courses and Credits Possible 

9th 

Students are in classes taught by high school 
staff serving as adjunct college instructors. 

During the academic year: 2 courses 

Total: up to 6 credits 

10th 

Students are taught by college instructors in 
classes that are only for Early College students. 

During the academic year: 2 courses 

During the summer: 1 course 

Total: up to 9 credits 

11th 

Students are in classes with traditional college 
students. Often by the junior year, dual 
enrollment courses make up 75% of a student's 
schedule. 

During the academic year: 4 courses 

During the summer: 1 courses 

Total: up to 21 credits 

12th Students are in classes with traditional college 

students. Often by the senior year, dual 
enrollment courses make up 100% of a 
student's schedule. 

SOURCE: A\R & SRI, 2009 

During the academic year: 4 courses 

During the summer: 2 courses 

Total: up to 24 credits 


Another component of college integration is location; Early Colleges opened on college campuses whenever 
possible. In this study, 8 of the 10 sites were located on a college campus. Often termed “the power of site,’’ the 
concept is that students respond to the high expectations and peer models that the college environment 
provides by taking more responsibility for their education. For Early Colleges located on a college campus, 
students have the opportunity to take dual enrollment courses predominantly taught by college faculty. Previous 
research also documented that Early Colleges located on college campuses had higher attendance and 
assessment proficiency rates compared to other Early Colleges. Students at Early Colleges located on college 
campuses reported more academic engagement and self-confidence, fewer disruptive behaviors among their 
peers, greater college credit accumulation, and higher post-Early-College educational aspirations than students 
at Early Colleges not located on a college campus (AIR & SRI, 2008, 2009). 
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Still, Early College students began accessing college earlier and enrolled in college during 
high school at significantly higher rates than their comparison peers. From prior research, 
we know that exposure to college instruction in high school is valuable for helping students 
understand the academic expectations of college coursework (AIR & SRI, 2006, 2007). Prior 
studies found that high school and college instructors varied in their instructional approach 
and the levels of support they provided to students. Whereas college professors presented 
information and placed the responsibility on students to learn the material, high school 
instructors emphasized the material’s relevance to students’ lives and provided students 
with more support for learning the material (Duffy, Cassidy, Keating, & Berger, 2009). Prior 
work has also found that college professors did not give Early College students any special 
treatment (AIR & SRI, 2006) and that students reported appreciating being treated “more 
like adults” in their college courses, relative to their high school courses (AIR & SRI, 2007). 

Consistent with these prior findings, in students’ open-ended survey responses, many 
reflected on how college exposure in high school prepared them academically for college 
after high school. For example, one Early College student noted, “I felt that being able to 
start taking college courses at [Early College] while still in high school made it much easier 
to adapt from high school to full-time college student after graduation.” Another Early 
College student echoed this: “Being able to take college classes in high school was a 
wonderful opportunity for me. I felt much more prepared than my peers when I came to 
college and coursework got harder.” 

Student survey responses also illustrated that being exposed to college in high school gave 
them a head start in acclimating to the increased freedom and time management skills 
required of college students. One Early College student said, “My high school was located 
on [a] university campus so it was a really great experience for me. I walked around with 
other college students and had access to the university’s buildings, and it felt liberating.” 
Another Early College student stated, “I had a great time enrolled in high school and college. 
Being immersed in both cultures simultaneously made me a better multitasker and forced 
me to grow up.” 

College Credit Accrual 

Of course, it is not just exposure to college courses but also college credit accrual that puts 
students on an accelerated path to postsecondary degree attainment. Early College had a 
statistically significant positive impact on earning college credits in high school, with 
66.3 percent of Early College students and 21.2 percent of comparison students earning any 
college credit (see Exhibit 4.2). Additionally, Early College students were significantly more 
likely to meet the initiative’s goal of earning at least one year of college credit. As Exhibit 4.2 
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shows, almost half (49.1 percent) of Early College students earned at least one year of 
college credit, relative to 5.5 percent of comparison students. 

Exhibit 4.2. Percentage of Students Who Earned College Credit in High School, by Study Group 



n = 1,294 students (724 Early College, 570 comparison) 

SOURCE: Student survey data from 2011-12 

NOTES: Early College means are unadjusted means, and comparison group means were computed by subtracting estimated 
Early College effects from the unadjusted Early College means. 

* p < .05; ** p < .01; *** p < .001 

Overall, the rates of college enrollment and credit accrual for treatment students indicate 
that Early Colleges provided college exposure, as the initiative intends. See the sidebar, 
How Early Colleges Integrate College Courses , for additional details. 

In open-ended survey responses, Early College students commented on the importance of 
earning college credit while in high school to their later college progression. One Early 
College student noted, “The program of being at an Early College and taking so many dual 
credit courses along with actual college courses has given me the greatest opportunity to 
prepare myself for what the real world offers. Being already a sophomore in college barely 
graduating from high school is a great achievement.” Another Early College student said, 
“My experience at [Early College] is one that I will never forget. It paved the path to where I 
am today, enrolled in [University’s nursing program] at the age of 18. If I had not attended 
[Early College], I would be applying to the program two years from now. [Early College] 
helped me to discover my true potential.” 

Comparison students may not have had access to college courses, but they may have had 
access to Advanced Placement (AP) coursework and, thus, the opportunity to earn college 
credits by passing AP exams. We found that comparison students were significantly more 
likely than their Early College peers to enroll in an AP course in high school. As Exhibit 4.3 
shows, over half (55.5 percent) of comparison students took at least one AP course, 
whereas less than a quarter (21.8 percent) of Early College students took at least one AP 
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course in high school. Several comparison students who were enrolled in college after high 
school commented on how useful their AP courses were in preparing them for college. One 
comparison student said, “My AP English classes were really the only thing that remotely 
helped me prepare for college. Nothing else in high school was really helpful at all.” 

Exhibit 4.3. Percentage of Students Who Enrolled in at Least One AP Course and Passed at Least One 
AP Exam in High School, by Study Group 



n = 1,294 students (724 Early College, 570 comparison) 

SOURCE: Student survey data from 2011-12 

NOTES: Early College means are unadjusted means, and comparison group means were computed by subtracting estimated 
Early College effects from the unadjusted Early College means. 

* p < .05; ** p < .01; *** p < .001 

To earn college credit, students taking AP courses must pass the accompanying exam. As 
Exhibit 4.3 illustrates, comparison students were significantly more likely to pass at least one 
AP exam; 33.3 percent of comparison students compared with 8.5 percent of Early College 
students passed any AP exam. The Early College student rates of AP course-taking and 
exam passage may seem high considering their access to college courses. However, as 
noted previously, some treatment (Early College) students never attended the Early College 
and some transferred out; these students may have taken AP courses at their high schools. 
Additionally, some Early College students had access to AP courses. 

Although comparison students enrolled in AP courses and passed AP exams at significantly 
higher rates than Early College students, AP course-taking did not lead to substantial college 
credit accumulation. On average, Early College and comparison students passed less than 
one AP exam (0.23 AP exams passed for Early College students and 0.70 for comparison 
students); thus AP coursework did not meaningfully accelerate students’ path toward 
postsecondary degree attainment. 

As this section has shown, college exposure in high school helped Early College students to 
see college as a possibility and provided them with a head start toward postsecondary 
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degree attainment by allowing them to earn a substantial amount of college credit in high 
school. 

Academic Rigor 

Taking college courses most notably differentiates Early College students’ high school 
experience from their peers, but Early Colleges also are expected to offer rigorous high school 
and college courses. Completing a program of college-preparatory coursework is a traditional 
indicator of college readiness for high school students. High school transcript studies have 
found that academic curricular intensity, defined as the number of Carnegie units completed in 
English, mathematics, science, history, and foreign language as well as the number of 
Advanced Placement and remedial courses taken, is more predictive of bachelor’s degree 
attainment than high school GPA, class rank, or test scores (Adelman, 1999, 2006). 

Therefore, we analyzed several indicators of high school academics to understand whether 
there were differences in the rigor of students’ high school courses and academic preparation 
for college. 

Mathematics and science are two academic subjects with commonly defined college- 
preparatory courses. Although there are no national standards for college readiness, we 
defined a college-preparatory mathematics course sequence as algebra I, algebra II, and 
geometry. This sequence is specified by four of the five states in our study. We also 
analyzed students’ completion of an advanced mathematics course (either pre-calculus or 
calculus) as an indicator of students’ readiness for college mathematics. For science, we 
defined the college-preparatory course sequence as biology, chemistry, and physics, which 
is specified by three of the five study states. 

As Exhibit 4.4 shows, there was no significant difference between Early College and 
comparison students in completing a college-preparatory mathematics sequence or an 
advanced mathematics course. Overall, over three quarters of study students completed a 
college-preparatory mathematics sequence in high school (75.4 percent Early College, 78.1 
percent comparison). Additionally, almost 60 percent of students in both groups completed 
an advanced mathematics course in high school (58.2 percent Early College, 59.9 percent 
comparison). 

An impact study of North Carolina Early Colleges found that, in ninth grade, Early College 
students were significantly more likely than comparison students to take algebra I or a 
higher mathematics course (Edmunds, Bernstein, et al., 2012). However, despite the 
statistically significant difference, overall rates were high across groups; 90 percent of Early 
College students and 81 percent of control students took these courses in ninth grade 
(Edmunds, Bernstein, et al., 2012). In this study, students reported all of the mathematics 
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courses they took in 9th through 12th grade, including their high school and college courses. 
The studies used different measures, but both studies found that Early College and 
comparison students completed college preparatory mathematics courses at high rates. 

In science, however, Early Colleges had a statistically significant, positive impact on course¬ 
taking. Slightly over half (50.7 percent) of Early College students and 41.9 percent of 
comparison students completed a college preparatory science sequence (see Exhibit 4.4). 


Exhibit 4.4. Percentage of Students Who Completed College Preparatory Mathematics and Science 
Courses in High School, by Study Group 



College preparatory math Advanced math course College preparatory science 
sequence sequence 

High School Course-Taking 


■ Early College 

■ Comparison 


n = 1,294 students (724 Early College, 570 comparison) 

SOURCE: Student survey data from 2011-12 

NOTES: Early College means are unadjusted means, and comparison group means were computed by subtracting estimated 
Early College effects from the unadjusted Early College means. 

* p < .05; ** p < .01; *** p < .001 


The student survey also included items about instructional rigor (together forming an 
instructional rigor scale), which asked students to report the frequency with which they 
engaged in various activities in their high school courses, such as writing five or more pages 
on a topic, researching information, and explaining their thinking. Appendix A contains 
further information on these items and the scale. Early College had a positive impact on 
instructional rigor; Early College students reported engaging in rigorous learning activities 
significantly more frequently than did the comparison students, with means of 2.7 and 2.5, 
respectively, on a 0 to 4 scale, on which 0 indicates “never” and 4 indicates “almost every 
day” (effect size = 0.35; see Exhibit 4.5). On average, students in both groups reported 
engaging in rigorous academic activities several times per month. 
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Exhibit 4.5. Students’ Reported Frequency of Engaging in Rigorous Academic Activities in High School, 
by Study Group 
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n = 1,294 students (724 Early College, 570 comparison) 

SOURCE: Student survey data from 2011-12 

NOTES: Early College means are unadjusted means, and comparison group means were computed by subtracting estimated 
Early Colleqe effects from the unadjusted Early College means. Response options ranqed from “never” to “almost every day.” 
* p < .05; ** p < .01; *** p < .001 

Our survey measures provide only a glimpse into the academic rigor of Early College and 
comparison students’ high school experiences, which is one of the Early College curricular 
and instructional emphases. Students’ responses to the open-ended survey question 
supported the finding that Early College students perceived their high schools to be more 
rigorous than comparison students perceived theirs. One Early College student said about 
her school, “My high school experience was different than any other regular high school. It 
was all academics and it took so much more hard work than other teens our age. But even 
after hearing what it was like to go through ‘the best four years of your life’ at a regular high 
school, I wouldn't change it for anything else.” 


Supports 

In addition to offering students access to dual enrollment courses, supports are often 
required to help students adjust to the different norms and expectations at the college level 
(Cassidy, Keating, & Young, 2010). To address this need, Early Colleges are expected to 
combine college exposure and academic rigor with academic and social supports to help 
students manage college coursework and provide them with the skills required for college 
success and completion. There are a wide variety of supports that Early Colleges provide, 
from academic tutoring and advising to study skills, time management, self-advocacy, and 
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other “college life skills” (AIR & SRI, 2008, 2009; Cassidy et al., 2010). The supports 
examined in this study are not comprehensive, but they include several measures that prior 
work has identified as important features of Early Colleges—college-going culture, general 
college information, instructor support, and several specific college preparation supports 
(AIR & SRI, 2008, 2009; Duffy et al., 2009). This section explores whether Early College 
students received greater supports from their schools to prepare for college than their 
comparison peers. 

First, Early Colleges aim to establish an environment where all students feel they are 
expected to go to college. Our survey measured college-going culture as the extent to which 
teachers, principals, and students in their high school expected students to go to college. 
Appendix A contains additional information about this scale. As Exhibit 4.6 demonstrates, 
Early College students reported a significantly stronger college-going culture at their high 
schools than comparison students. The average score for Early College students was 3.5, 
compared with 3.3 for comparison students, on the 1 to 4 college-going culture scale on 
which 3 represents agreement and 4 represents strong agreement (effect size = 0.32). 


Exhibit 4.6. Students’ Reported College-Going Culture in High School, by Study Group 
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SOURCE: Student survey data from 2011-12 

NOTES: Early College means are unadjusted means, and comparison group means were computed by subtracting estimated 
Early College effects from the unadjusted Early College means. Response options ranged from “strongly disagree” to “strongly 
agree.” 

* p < .05; ** p < .01; *** p < .001 

Second, Early Colleges aim to support students in meeting the college-going expectations 
described above. One way Early Colleges do this is by talking about college and providing 
students with information about college. Our survey asked whether students had at least 
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one “very helpful” resource at their high school to provide them with general information 
about college. As Exhibit 4.7 shows, about 80 percent of students in both groups reported 
having a “very helpful” resource to learn about college (83.6 percent Early College 
compared with 79.5 percent comparison—a difference that was not statistically significant). 46 


Exhibit 4.7. Percentage of Students Who Reported at Least One “Very Helpful” Resource at Their High 
School for Providing General College Information, by Study Group 



n = 1,294 students (724 Early College, 570 comparison) 

SOURCE: Student survey data from 2011-12 

NOTES: Early College means are unadjusted means, and comparison group means were computed by subtracting estimated 
Early College effects from the unadjusted Early College means. 

* p < .05; ** p < .01; *** p < .001 

Third, Early Colleges aim to support students academically and socially. Prior research has 
shown that many Early Colleges in the initiative integrate features to promote personalized 
instruction, including teachers moving with their students to the next grade level and 
providing small-group and individualized instruction (AIR & SRI, 2007). The student survey 
measured instructor support as an indicator of the Early College emphases on 
personalization and instructor-student relationships to support students. The scale contained 
items asking the extent to which students agreed that their high school teachers cared about 
them and encouraged them, and other similar items. Appendix A contains additional 
information about this scale. Both groups agreed that their high school instructors were 
supportive, but the level of support was significantly stronger for Early College students. 

As Exhibit 4.8 illustrates, Early College students had a mean score of 3.4 and comparison 


46 We also examined whether students received general college information from resources outside of school (e.g., family, 
friends). About 40 percent of both groups reported having at least one “very helpful” resource for providing them with general 
college information outside of school (38.8 percent Early College, 40.0 percent comparison; a statistically insignificant 
difference). 
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students had a mean score of 3.2 on the instructor support scale, on which 3 represents 
agreement and 4 represents strong agreement (effect size = 0.31). 


Exhibit 4.8. Students’ Reported Instructor Support in High School, by Study Group 


■ Early College 

■ Comparison 


Instructor support 



n = 1,294 students (724 Early College, 570 comparison) 

SOURCE: Student survey data from 2011-12 

NOTES: Early College means are unadjusted means, and comparison group means were computed by subtracting estimated 
Early College effects from the unadjusted Early College means. Response options ranged from “strongly disagree” to “strongly 
agree.” 

* p < .05; ** p < .01; *** p < .001 

On the survey, Early College students noted the support they received from their high 
school teachers. One said, “The teachers were always there to help and [took] the time to 
give us information about college. We had complete access to the faculty at all times.” 
Another commented, “My teachers were driven to make sure we succeeded in achieving 
both [an associate's degree and a high school diploma] and in pursuing a higher version of 
education.” 


Fourth, Early Colleges aim to provide specific supports for students to prepare for and apply 
to college. Prior research found that Early Colleges provide a variety of college preparation 
supports, including college preparation and support courses, college information sessions, 
college entrance exam preparation, college application help, financial aid application help, 
college tours, and career guidance (AIR & SRI, 2008, 2009). In 2008-09, 89 percent of 
surveyed Early Colleges reported providing formal academic or social support classes or 
seminars (AIR & SRI, 2009). 47 In 2006-07, 59 percent of the surveyed Early Colleges 
reported having support classes with a goal of exposing students to college and 


47 See Hindo, Barnett, and Kim (2010) for a description of Early College seminars devoted to supporting college courses. 
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employment topics (AIR & SRI, 2008). For example, one Early College class was called 
“Apply for College.” 

Our student survey asked about high school supports for three college preparation activities: 
preparing for the ACT or SAT, completing college applications, and completing financial aid 
applications. The analyses of these measures were based on a non-random subsample. For 
example, students were asked about the sources and quality of help they received preparing 
for the ACT or SAT only if they reported taking the ACT or SAT exam. 

Exhibit 4.9 shows that Early College and comparison students did not differ significantly in 
their rates of participation in each of three activities we asked about on the survey. Among 
both Early College and comparison students, about 80 percent took the ACT or SAT, about 
80 percent applied for college, and about 70 percent applied for financial aid. The similar 
percentages of students in both groups suggest that it is unlikely that the subset of Early 
College students and comparison students included in the college preparation support 
analyses were systematically different from the overall sample. Nevertheless, this 
subsample was not randomly selected; results from the following analyses should be 
interpreted with caution. 


Exhibit 4.9. Student Participation in College Preparation Activities, by Study Group 



College Preparation Activity 


■ Early College 

■ Comparison 


n = 1,294 students (724 Early College, 570 comparison) 

SOURCE: Student survey data from 2011-12 

NOTES: Early College means are unadjusted means, and comparison group means were computed by subtracting estimated 
Early College effects from the unadjusted Early College means. 

* p < .05; ** p < .01; *** p < .001 

For each of the above activities, the survey asked students about supports provided in 
school and outside of school, resulting in a total of six college preparation support 
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measures. Focusing on the quality of support, each measure indicates whether the student 
reported having at least one “very helpful” resource (e.g., teacher, counselor) for the 
specified college preparation activity (see Appendix A for more information). Exhibit 4.10 
shows that there were significant differences between Early College and comparison students in 
two of the three college preparation supports at their high school. Three quarters (75.0 percent) 
of Early College students reported having at least one very helpful resource at their high school 
to help them with college applications, versus 66.1 percent of comparison students (effect size = 
0.26). Further, 63.1 percent of Early College students reported having at least one helpful 
resource at their high school to help them with financial aid forms, versus 55.3 percent of 
comparison students (effect size = 0.20). These findings are consistent with prior research, 
which found that Early Colleges provide systematic college and financial aid application 
supports (AIR & SRI, 2009). Flowever, students’ reports of receiving help preparing for the ACT 
or SAT were about the same for the Early College and comparison groups. 48 


Exhibit 4.10. Percentage of Students Who Reported at Least One “Very Helpful” Resource at Their High 
School, by College Preparation Activity and Study Group 
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The sample sizes vary by item because students were only asked about the supports they received for each activity they 
reported completing: SAT/ACT help r? = 1,053 (596 Early College, 457 comparison); College application help n = 1,067 (601 
Early College, 466 comparison); Financial aid help n = 890 (512 Early College, 378 comparison). 

SOURCE: Student survey data from 2011-12 

NOTES: Early College means are unadjusted means, and comparison group means were computed by subtracting estimated 
Early College effects from the unadjusted Early College means. 

* p < .05; ** p < .01; *** p < .001 


48 In addition to examining ACT/SAT-taking, we examined ACT/SAT scores as an indicator of college readiness. Since 
students could take either the ACT or SAT or both exams, we converted students’ highest reported SAT score into ACT units 
and used students’ highest overall exam score (ACT, n.d.). Using ACT units, Early College students scored an average of 23 
and comparison students scored an average of 22—not a significant difference. 
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Although Early College students received significantly stronger college and financial aid 
application supports from their school than comparison students, it is possible that 
comparison students may have found similar help elsewhere. Therefore, we examined 
support for college preparation activities that students found outside of school. As Exhibit 
4.11 shows, the percentage of students reporting at least one “very helpful” resource outside 
of school varied across specific college preparation activities. However, similar percentages 
of Early College and comparison students reported help with each activity; differences were 
not statistically significant. 


Exhibit 4.11. Percentage of Students Who Reported at Least One “Very Helpful” Resource Outside of 
School, by College Preparation Activity and Study Group 



Help preparing for the Help completing college Help completing financial 
ACT/SAT applications aid forms 


College Preparation Activities 


■ Early College 

■ Comparison 


The sample sizes vary by item because students were only asked about the supports they received for each activity they 
reported completing: ACT/SAT help n = 1,053 (596 Early College, 457 comparison); College application help n = 1,067 (601 
Early College, 466 comparison); Financial aid help n = 890 (512 Early College, 378 comparison). 

SOURCE: Student survey data from 2011-12 

NOTES: Early College means are unadjusted means, and comparison group means were computed by subtracting estimated 
Early College effects from the unadjusted Early College means. 

* p < .05; ** p < .01; *** p < .001 

The survey data indicated that both Early College and comparison students took steps to 
prepare for college and both groups had access to supports. However, Early College students 
experienced significantly higher college-going expectations and felt more supported by 
instructors. Additionally, there is some evidence to suggest that Early College students 
received stronger support at their high school to complete college applications and apply for 
financial aid than comparison students. The sentiment of several comparison students was 
reflected in this statement by one student: “I was left to figure out applying for colleges by 
myself.” The college and financial aid application process is difficult to navigate, and school 
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supports are helpful, especially for first-generation college-goers and disadvantaged students 
who are the initiative’s target population and who often do not have help from other resources. 
As one student who attended an Early College but transferred to another school said, “I had 
no help other than a family member who did not have college experience but wanted me to 
attend college, so no one around me knew how to even apply.” Thus, college application 
supports may distinguish Early Colleges from other high schools. 

Conclusion 

As this chapter has demonstrated, being admitted to an Early College had a statistically 
significant impact on students’ high school experiences. First, Early College students earned 
significantly more college credits than their comparison peers, putting them on a faster track 
to college degree attainment. Second, Early College students reported a significantly more 
rigorous high school academic experience than comparison students did. Third, Early 
College students reported significantly stronger college-going culture and instructor support 
than comparison students. Fourth, there is evidence to suggest that Early College students 
received stronger college application and financial aid application supports than their 
comparison peers. However, as noted above, findings about specific college preparation 
supports only apply to a non-random subset of the survey sample. 

The high school experiences and supports measured in this study were limited due to the 
retrospective nature of the survey. They do not represent the full range of students’ high 
school experiences or the comprehensive set of academic and social supports that Early 
Colleges provide. Nevertheless, these findings suggest that a distinguishing feature of the 
Early College model is providing systematic supports to help students make a successful 
transition to college. As a whole, the measures in this chapter provide context to the 
outcome differences discussed in the previous chapter, illustrating that Early Colleges had a 
positive and meaningful impact on students’ college exposure, college credit accrual, and 
supports in high school. 
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Chapter 5. Summary 

The ECHSI, initiated by the Bill & Melinda Gates Foundation, has been growing for over a 
decade. Between 2002 and 2012, over 250 Early Colleges opened as part of this initiative, 
and many more Early Colleges unaffiliated with the ECHSI have opened nationwide (Jobs 
for the Future, 2012). AIR and SRI studied the ECHSI for seven years (2002-09), 
documenting growth and sustainability. By 2009, Early Colleges appeared to be working 
well for many students (AIR & SRI, 2008, 2009). Early Colleges nationally were serving 
primarily minority and low-income students. Early Colleges achieved higher assessment 
proficiency rates and higher graduation rates than other area high schools. Most importantly, 
Early College graduates earned between a semester and a year of college credits while in 
high school. 

By 2009, evidence was promising, but many questions remained. Most of the evidence from 
the program focused on high school outcomes since there were few high school graduates 
during the early years of the initiative. Early Colleges reported on the college plans of 
students and students reported on the amount of college credits they would accrue by 
graduation, but the question remained: What are the long-term outcomes for Early College 
students? 

In addition, the prior research evidence on Early Colleges was largely descriptive. Early 
Colleges could have been recruiting higher performing students. Students who chose to 
enroll could have been more academically motivated and engaged. A descriptive 
examination cannot fully disentangle the influence of these factors from the causal impact of 
Early Colleges on students, prompting another question: What is the impact of Early 
Colleges on students? 

Finally, although the overall findings from previous research were positive, evidence 
indicated that some types of students benefited more from attending an Early College than 
others. For example, more females than males seemed to stay enrolled in high school. In 
addition, minority students, low-income students, and first-generation college-going students 
reported lower GPAs than non-minority students, high-income students, and students who 
were not first-generation college-goers. Thus, prompting a final question: Does the impact of 
Early Colleges systematically differ for different groups of students? 

Buoyed by the positive findings from the initial evaluation work and inspired by these 
remaining questions, we embarked on a study to determine the program’s impact on both 
students’ high school and postsecondary outcomes. Our two guiding research questions 
were as follows: 
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1. Do Early College students have better outcomes than they would have had at other 
high schools? 

2. Does the impact of Early Colleges vary by student background characteristics? 

We used existing lottery admission records to compare the educational outcomes for 
students admitted to Early Colleges by lottery to the educational outcomes for students who 
were not admitted by lottery. Because the lottery-based randomization ensured that the 
Early College and comparison students were similar at high school entry, we can draw 
strong conclusions about impact. By examining data from schools, districts, states, the NSC, 
and students, we can provide strong evidence related to these research questions. 

Early College Impact on Outcomes 

Do Early College students have better outcomes than they would have had at other 
high schools? 

In high school, Early College students had significantly higher ELA assessment scores and 
graduation rates than comparison students. However, the groups did not differ significantly 
on high school mathematics scores or GPA. 

In terms of college enrollment and degree attainment, Early Colleges provided students a 
large advantage. Early College students were more likely than comparison students to have 
enrolled in college. They were also more likely to have earned a college degree. Because 
Early College students start college in high school, we should expect Early College students 
to enroll in college and complete college degrees earlier. However, both of these findings 
held for every point in time we examined. In other words, there was no evidence of 
comparison students “catching up” to Early College students in later years. 

Almost 20 percent of Early College students earned a college degree, usually an associate’s 
degree, by the end of the traditional high school years. These findings are consistent with 
other recent research on Early Colleges. A study of North Carolina Early Colleges found that 
about one quarter of Early College students earned a college degree within two years of 
high school (Edmunds, Unlu, Glennie, Bernstein, & Smith, 2013). Similarly, Jobs for the 
Future (2013) found that at established Early Colleges nationally, about one quarter of 
graduates obtained a college degree while in high school. 

Early College students were significantly more likely to enroll in a four-year college than 
comparison students, both during and after high school. Although most Early College 
students who started college while in high school enrolled in a two-year college, most 
students who enrolled after high school enrolled in a four-year college. These findings 
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suggest that Early Colleges set students on a different academic trajectory. Early Colleges 
were not diverting students away from four-year colleges. 

Differential Early College Impact 

Does the impact of Early Colleges vary by student background characteristics? 

We found that the effects of Early College were generally similar for students with different 
background characteristics, but when the effects differed, the difference was generally in 
favor of underrepresented groups. The Early College impact on high school graduation and 
college enrollment was similar for students with different background characteristics. 49 For 
college degree attainment, Early Colleges not only had a significant overall impact on all 
students, but also had significantly stronger impacts for the traditionally underrepresented 
student groups than for their peers. 

The Early College impact on degree attainment was stronger for 

■ minority students relative to non-minority students, and 

■ low-income students relative to higher income students. 

Given that research has shown that, generally, minority and low-income students are less 
likely to obtain a college degree relative to their peers (NCES, 2012b and 2012c), our 
findings about the stronger Early College impact on minority and low-income students’ 
college degree attainment suggest that Early Colleges reduced the commonly observed 
gaps in degree attainment related to minority status and low-income status. In Texas, Struhl 
and Vargas (2012) also found that participation in dual enrollment programs increased the 
likelihood of college degree attainment for all students regardless of race, ethnicity, or 
income. Of particular note is that, on average, minority Early College students were at least 
as likely as non-minority Early College students to have earned a college degree. 

On the other hand, the Early College impact on college degree attainment was stronger for 

■ female students relative to male students, and 

■ students with higher levels of prior achievement relative to students with lower levels 
of prior achievement in ELA and mathematics. 


49 In this study, we only examined differences in Early College impacts based on student characteristics for three key 
outcomes. It is possible that the Early College impacts on other outcomes may differ for students with different characteristics. 
Struhl and Vargas (2013), for example, found that the positive effect of participation in dual enrollment programs on the 
likelihood of attending a four-year institution was stronger for low-income students than for other students. 
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Female and higher-achieving students are typically more likely to obtain a college degree 
(NCES, 2012g; Adelman, 2006). These results suggest that Early Colleges may have 
actually widened the gaps in degree attainment related to gender and prior achievement. 

Taken together, these results show that Early Colleges did not leave underrepresented 
students behind. The goal of the ECHSI is to reach students from underrepresented 
populations and make college completion a common outcome. The Early Colleges included 
in this study certainly achieved the intended impacts for underrepresented groups. 

A note of caution should be observed when considering the impact of Early Colleges on the 
outcomes for traditionally underrepresented student groups. Although the goal of the ECHSI 
is to reach students from underrepresented populations, these Early Colleges attracted 
more academically prepared and academically ambitious students than typical district 
students. Overall, both Early College and comparison students had impressive high school 
graduation rates, exceeding 80 percent. By comparison, in one large urban district where a 
large number of the study students attended high school, the estimated four-year high 
school graduation rate in 2009 was just over 50 percent (EPE Research Center, 2012). 
College enrollment rates for students in the study sample were quite high as well—over 70 
percent. By comparison, only 40 percent of 18-to-24-year-olds in the nation enrolled in 
college in 2010 (NCES, 2012h). 

Therefore, although this study found that the impact of Early Colleges on degree completion 
was stronger for traditionally underrepresented students, it is important to note that, because 
our sample comprises lottery participants in selected sites, it is not representative of the 
population of underrepresented students in America. 

Early College Impact on High School Experiences 

As expected, the high school experiences for Early College students differed significantly 
from those of comparison students. 

College Exposure - Students admitted to Early Colleges received significantly more 
exposure to college in high school than did their peers in the comparison group. About half 
of the Early College students earned at least one year of college credit while in high school. 
Although comparison students also had access to college classes, they accumulated 
significantly fewer college credits. 

Rigorous Academics -When the ECHSI was starting, observers and stakeholders 
wondered if the schools could provide rigorous academics and introduce college courses. 
The fear was that the pressure to take college classes would lead students into easy college 
classes and away from rigorous high school classes. The evidence suggests that this did 
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not occur. Early College students engaged in rigorous courses and activities at the same 
level as, or at an even higher level than, the comparison students. 

Supports- Our evidence indicates that study students felt well supported in high school. 
Generally, they felt that their peers and teachers wanted them to go to college, their 
teachers were there to help them, and their schools provided helpful supports in applying for 
college. Although Early College students felt more strongly supported than comparison 
students, the general message is that regardless of where they went to school, most of 
these students had access to supports. 

Caveats and Future Research 

This study provides strong evidence of the positive impact of Early College on students. 
However, several caveats are worth repeating. First, these findings are only applicable to 
the 10 Early Colleges included in this study. During the study years, there were over 150 
Early Colleges open across the country and many of them had been open for at least five 
years. There are many Early Colleges that these 10 sites may not represent. For example, 
the study Early Colleges may have been particularly effective, gaining a reputation that led 
to their oversubscription, and, hence, a lottery for admission. Alternatively, they may have 
been well managed, as evidenced by their keeping of lottery records for many years. 

Second, these findings are only applicable to the study years. These Early Colleges may 
have improved or worsened over time. Earlier research (AIR & SRI, 2008) suggested that 
student outcomes were stronger in the first years of an Early College’s opening. 

Third, these findings are only applicable to lottery participants. Many students apply to Early 
Colleges and are accepted without going through a lottery. For example, some Early 
Colleges automatically enrolled students with certain assessment scores. Students who did 
not participate in the lottery are not represented by this study. 

Fourth, these results are based on the premise that the lotteries resulted in random 
assignment of students to attend or not attend an Early College. As we have noted, our 
extensive research turned up only minor concerns over lottery implementation in two sites, 
and even with those sites excluded, the findings do not change much. However, we did not 
run the lotteries, and we found a significant baseline difference between Early College and 
comparison students in one baseline student characteristic—student ELA test scores. Given 
that there was no significant baseline difference in mathematics scores, the observed 
difference in mathematics scores was in the opposite direction of the difference in ELA, and 
there were no significant differences in student demographics, we believe the lotteries were 
fair. But there could have been procedures that resulted in some students not being 
randomly assigned. 
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These caveats suggest that a representative impact study would be highly desirable. One 
approach could be to randomly select Early Colleges from the ECHSI and then follow 
students prospectively after random assignment. A study of Early Colleges in North Carolina 
is a step in this direction (Edmunds, Bernstein, et al., 2012), but it does not include a 
representative set of Early Colleges or students. Although potentially enlightening, this type 
of study would certainly be a challenge, as not all schools are oversubscribed, and not all 
schools use a lottery for admission. 

Another important avenue for research involves following students for a longer period of 
time. This study followed students through up to three years after high school. However, 
many statistics on postsecondary attainment allow four years after high school to attain an 
associate’s degree and six years to attain a bachelor’s degree (NCES, 2012i). If additional 
research were to follow students over a longer period of time, we hypothesize that, that the 
following will be true for Early College students relative to comparison students: 

■ Early College students will accrue less educational debt. Early College students 
accrued significant college credits during high school at no or reduced cost. If they 
spend less time in college after high school, they should finish their education with 
less debt. 

■ Early College students will have an advantage in finding employment. Earning a 
degree may provide advantages to Early College students when competing for a job 
against others with weaker educational qualifications. In addition, earning a degree 
earlier may provide advantages to Early College students when competing against 
others with the same qualifications: if a student graduates earlier than other students, 
it may signal to potential employers that the student is more ambitious and 
academically successful than other applicants with the same credentials. 50 

■ Early College students will have higher lifetime earnings. If Early College students 
join the workforce earlier than their peers with the same education level, and are 
more competitive in obtaining jobs, then the Early College is increasing their potential 
for lifetime earnings. 

Finally, these benefits may go beyond the students and their families. An earlier study 
estimated the return on investment based on the preliminary assumption that about 48 
percent of Early College students and 20 percent of traditional students eventually obtain an 
associate’s or bachelor’s degree (Palaich, Augenblick, & Maloney, 2007; Vargas, 2013). 
Under these assumptions, which are appropriate based on the impacts reported for this 
study, the authors found long-term, financial benefits both for Early College students and 


50 See, for example, Flores-Lagunes and Light (2010), which indicates that among college degree holders, longer completion 
times have negative workforce outcomes. 
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their families and for educational institutions and governments. Therefore, a promising line 
of future research is to examine the impact of Early Colleges on longer-term outcomes. 

In this study, we only scratched the surface in examining the treatment, or the “black box.” 
We hope that future research will investigate the high school experiences more deeply and 
test the conceptual framework. Additionally, studies including a larger number of sites may 
allow researchers to examine the variation in Early College impact, and factors that may 
relate to the variation in impact. For example, early descriptive research suggested that 
Early Colleges on college campuses provided greater benefits to students than other Early 
Colleges (AIR and SRI, 2009). This could be tested by comparing the Early College impact 
for Early Colleges located on a college campus with the impact for Early Colleges located on 
a high school campus in a larger-scale multisite impact study. 

Finally, this research inspires several questions about policies that might support Early 
College. To replicate these results, must schools implement the full Early College model, or 
can high schools leverage current dual enrollment policies to improve student access and 
success in college? What supports are provided by Early Colleges, and how do they relate 
to student success in college? What is the role of state policy in the success of Early 
Colleges and dual enrollment programs? 51 Can online college coursework be successfully 
used to reform high school education? 

Conclusion 

The Early Colleges in this study yielded significant and meaningful improvements in student 
outcomes. We believe the success of the Early College model lies in the relationship 
between high school and college and its vision for educating young adults. This initiative 
challenges long-held assumptions about who can be successful in college and when. By 
combining two systems, secondary and postsecondary, it provides a different way of 
thinking about the role, and potential, of college. The ECHSI challenges the artificial 
separation between high school and college. The initiative completely reengineers the 
educational pathway, and many students have experienced greater success following this 
new path. 

We conclude the report with the words of two successful Early College students. Clinton 
earned an associate’s degree by the time he graduated from his Early College. 52 After high 
school graduation, he enrolled in a “highly competitive” four-year college and by the end of 


51 For example, Alabama is working toward allowing any high school student to take community college courses for free 
(McClure, 2013). 

2 All names have been changed to protect student confidentiality. 
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our study he was in his third year of college. 53 Clinton said, “My time at [the Early College] 
has prepared me for schoolwork at a higher level because I was basically performing the 
same level of work throughout high school. The environment pushed me to take every 
opportunity offered to me... I am a better student for having attended [Early College].” For 
Clinton, the Early College provided the preparation he needed to succeed at an 
academically demanding university. 

Alexia enrolled in a “very competitive” four-year college after attaining an associate’s degree 
in high school. By the end of this study, she was enrolled for her second year of college after 
high school. She reflected on the Early College’s effect on accelerating her education and 
her readiness for college: “[The Early College] allowed me to receive my associate’s degree 
nearly for free. Without [the Early College], I would not have my competitive edge that I have 
now at [college]; and I wouldn't have the same confidence, college smarts, or goals.” 
According to Alexia, attending an Early College not only gave her the opportunity to earn a 
college credential at little cost but also gave her the skills to help her succeed at the 
competitive college. As both of these students demonstrate, Early Colleges had an 
important impact on their educational outcomes and personal development. These students 
exemplify the difference that Early Colleges made in the lives of students. 


53 To protect the identity of students, we include the selectivity of colleges using the Barron's ranking system (Barron’s Profiles 
of American Colleges, 2013) instead of using the institutions’ names. 
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Appendix A: Measures 


Exhibit A.1 lists all variables that were used in this report, including their descriptions and 
their data sources: an administrative source (educational record data from schools, districts, 
states, and the National Student Clearinghouse [NSC]) or the student survey. The exhibit is 
organized by type of measure: student background characteristics, high school outcomes, 
college outcomes, and high school experiences. 


Exhibit A.1. Student Background Characteristic Measures, Descriptions, and Data Sources 


Measure 

Description 

Data Source 

Female 

Dichotomous indicator of student gender 

Administrative & 
Survey 

Minority 

Dichotomous indicator of whether the student is African American, 

Hispanic, Asian, Pacific Islander, Native American, multi-racial, or other 
minority group 

Administrative & 
Survey 

First-generation 

college-going 

Dichotomous indicator of whether either of the student's parents attended 
college 

Survey 

Low income 

Dichotomous indicator of whether student was eligible for the free or 
reduced-price lunch program, typically in eighth grade 

Administrative 

English language 
learner (ELL) 

Dichotomous indicator of whether the student was identified as an English 
language learner, typically in eighth grade 

Administrative 

Individualized 

Education Plan (IEP) 

Dichotomous indicator of whether student had an Individualized Education 
Plan, typically in eighth grade 

Administrative 

Prior achievement in 

ELA 

Standardized test score (z score) in English language arts (ELA) prior to 
entering high school, typically from eighth grade; standardized using the 
state mean and standard deviation for each year and grade level 

Administrative 

Prior achievement in 
mathematics 

Standardized test score (z score) in mathematics prior to entering high 
school, typically from eighth grade; standardized using the state mean and 
standard deviation for each year and grade level 

Administrative 
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Exhibit A.2. High School Outcome Measures, Descriptions, and Data Sources 


Measure 

Description 

Data Source 

High School Achievement 

Achievement in ELA 

Standardized test score (z score) in high school English language arts 
(ELA), typically from 10th grade; standardized using the state mean and 
standard deviation for each year and grade level 

Administrative 

Achievement in 
mathematics 

Standardized test score (z score) in high school mathematics, typically 
from 10th grade; standardized using the state mean and standard deviation 
for each year and grade level 

Administrative 

High school GPA 

A 0-4 variable indicating students' high school grade point average 

Survey 

Persistence 

Withdrawal from high 
school 

Dichotomous indicator of whether student withdrew from initial high school 
in which enrolled; withdrawals include transfers, dropouts, students 
expelled or incarcerated, and other withdrawal reasons 

Administrative 

Drop out 

Dichotomous indicator of whether student dropped out of high school 

Administrative 

Local transfer 

Dichotomous indicator of whether student withdrew from initial high school 
to transfer to another local option, including to attend other local public or 
private schools or for home schooling 

Administrative 

Graduation 

High school 
graduation 3 

Dichotomous indicator of whether student ever completed high school with 
a diploma or General Educational Development (GED) certificate 

Administrative 


a Fewer than five students in the study sample received GED certificates. 
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Exhibit A.3. College Outcome Measures, Descriptions, and Data Sources 


Measure 

Description 

Data Source 

College Enrollment 

Ever enrolled in 
college 

Dichotomous indicator of whether a student had ever enrolled in college 

For Cohort 1, data cover from May 2005 through October 2012 (Year 1 
through Year 7). For Cohort 2, data cover from May 2006 through October 
2012 (Year 1 through Year 6). For Cohort 3, data cover from May 2007 
through October 2012 (Year 1 through Year 5). 

NSC 

Enrolled in college by 
Year4 a 

Dichotomous indicator of whether a student enrolled in college by July of 
Year 4 after starting ninth grade 

NSC 

Enrolled in college by 
Year 5 a 

Dichotomous indicator of whether a student enrolled in college by July of 
Year 5 after starting ninth grade 

NSC 

Enrolled by Year 4 for 
2005-06 and 2006-07 
Cohorts 

Dichotomous indicator of whether a student enrolled in college by July of 
Year 4 after starting ninth grade for 2005-06 and 2006-07 Cohorts 

NSC 

Enrolled by Year 5 for 
2005-06 and 2006-07 
Cohorts 

Dichotomous indicators of whether a student enrolled in college by July of 
Year 5 after starting ninth grade for 2005—06 and 2006-07 Cohorts 

NSC 

Enrolled by Year 6 for 
2005-06 and 2006-07 
Cohorts 

Dichotomous indicators of whether a student enrolled in college by July of 
Year 6 after starting ninth grade for 2005-06 and 2006-07 Cohorts 

NSC 

Ever enrolled in a 2- 
year college 

Dichotomous indicator of whether a student ever enrolled in a 2-year 
college 

NSC 

Enrolled in a 2-year 
college by Year 4 

Dichotomous indicator of whether a student enrolled in a 2-year college by 
July of Year 4 after starting ninth grade 

NSC 

Enrolled in a 2-year 
college by Year 5 

Dichotomous indicator of whether a student enrolled in a 2-year college by 
July of Year 5 after starting ninth grade 

NSC 

Ever enrolled in a 4- 
year college 

Dichotomous indicator of whether a student ever enrolled in a 4-year 
college 

NSC 

Enrolled in a 4-year 
college by Year 4 

Dichotomous indicator of whether a student enrolled in a 4-year college by 
July of Year 4 after starting ninth grade 

NSC 

Enrolled in a 4-year 
college by Year 5 

Dichotomous indicator of whether a student enrolled in a 4-year college by 
July of Year 5 after starting ninth grade 

NSC 

Enrolled in partner 
institutions 

Dichotomous indicator of whether a student ever enrolled in the partner 
institution after high school graduation date 

Administrative & 

NSC 

Enrolled in college 
between Year 5 and 
Year 7 at any college 

Dichotomous indicator of whether a student enrolled between July of Year 

5 and October of Year 7 after starting ninth grade at any college 

NSC 

Enrolled at any 
college after high 
school graduation 
date 

Dichotomous indicator of whether a student enrolled at any college after 
date of high school graduation 

Administrative & 

NSC 
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Exhibit A.3. College Outcome Measures, Descriptions, and Data Sources (cont.) 



Enrolled in college Dichotomous indicator of whether a student enrolled between July of Year NSC 

between Year 5 and 5 and October of Year 7 after starting ninth grade at a 2-year college 

Year 7 at a 2-year 
college 

Enrolled in college Dichotomous indicator of whether a student enrolled between August of NSC 

between Year 5 and Year 5 and October of Year 7 after starting ninth grade at a 4-year college 

Year 7 at a 4-year 
college 


College Degree Attainment 


Any postsecondary Dichotomous indicator of whether a student ever received any type of NSC 

degree 15 postsecondary credential including a certificate, associate degree, or 

bachelor degree 

For Cohort 1, data cover from May 2005 through October 2012 (Year 1 
through Year 7). For Cohort 2, data cover from May 2006 through October 
2012 (Year 1 through Year 6). For Cohort 3, data cover from May 2007 
through October 2012 (Year 1 through Year 5). 


Ever earned a Dichotomous indicator of whether a student ever received a postsecondary NSC 

certificate certificate 

Ever earned A.A. Dichotomous indicator of whether a student ever received an associate's NSC 

degree degree 

Ever earned B.A. Dichotomous indicator of whether a student ever received a bachelor's NSC 

degree degree 

Postsecondary degree Dichotomous indicator of whether a student received any type of NSC 

earned by Year 4 postsecondary credential by the end of August of Year 4 after starting ninth 
grade 

Postsecondary degree Dichotomous indicator of whether a student received any type of NSC 

earned by Year 5 postsecondary credential by the end of August of Year 5 after starting ninth 
grade 

Postsecondary degree Dichotomous indicator of whether a student in the 2005-06 or 2006-07 NSC 

earned by Year 4 for cohort received any type of postsecondary credential by the end of August 
2005-06 and 2006-07 of Year 4 after starting ninth grade 
cohorts 

Postsecondary degree Dichotomous indicator of whether a student in the 2005-06 or 2006-07 NSC 

earned by Year 5 for cohort received any type of postsecondary credential by the end of August 
2005-06 and 2006-07 of Year 5 after starting ninth grade 
cohorts 

Postsecondary degree Dichotomous indicator of whether a student in the 2005-06 or 2006-07 NSC 

earned by Year 6 for cohort received any type of postsecondary credential by the end of August 
2005-06 and 2006-07 of Year 6 after starting ninth grade 
cohorts 
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Exhibit A.3. College Outcome Measures, Descriptions, and Data Sources (cont.) 



Earned any type of Dichotomous indicator of whether a student received any type of NSC 

postsecondary degree postsecondary credential between July of Year 5 and October of Year 7 
between Year 5 and after starting ninth grade 
Year 7 

Earned any type of Dichotomous indicator of whether a student received any type of Administrative & 

postsecondary degree postsecondary credential after date of graduation from high school nsc 

after high school 
graduation date 

Earned A.A. degree Dichotomous indicator of whether a student received an associate's degree NSC 

between Year 5 and between July of Year 5 and October of Year 7 after starting ninth grade 

Year 7 

Earned a B.A. degree Dichotomous indicator of whether a student received a bachelor's degree NSC 


between Year 5 and between July of Year 5 and October of Year 7 after starting ninth grade 
Year 7 


College Performance 


Developmental Dichotomous indicator of whether a student enrolled in any developmental Survey 

coursework 0 courses in college after high school 

College GPA d A 0-4 variable indicating students' college grade point average after high Survey 

school 

Perceptions of A 1-4 scale measuring students' level of difficulty keeping up with Survey 

academic difficulty in schoolwork, reading college materials, and writing papers (1 = very easy 

college d and 4 = very difficult); reliability (a) = 0.66 (scale included three items) 

Persistence in college Dichotomous indicator of whether a student was enrolled in college on NSC 

October 1 in Year 5 and Year 6 after starting ninth grade 

a Year 4 corresponds to academic year 2008-09 for Cohort 1,2009-10 for Cohort 2, and 2010-11 for Cohort 3. Year 5 
corresponds to academic year 2009-10 for Cohort 1, 2010-11 for Cohort 2, and 2011-12 for Cohort 3. See Exhibit 1.1 for a full 
correspondence between cohort years and academic years. 

b In this study sample, 13 Early College students and 9 comparison students earned certificates. 

c This item was administered only to students who, at the time of survey administration, reported ever enrolling in college after 
high school. 

d These items were administered only to students who, at the time of survey administration, reported completing at least one 
term of college after high school. 


American Institutes for Research 


Early College High School Initiative Impact Study 


86 

















APPENDIX A: MEASURES 


Exhibit A.4. High School Experience Measures, Descriptions, and Data Sources 


Measure 

Description 

Data Source 

College Exposure 

Enrolled in college in 

9th grade 

Dichotomous indicator of whether a student enrolled in college between 
August and July of Year 1 after starting ninth grade 

NSC 

Enrolled in college in 

10th grade 

Dichotomous indicator of whether a student enrolled in college between 
August and July of Year 2 after starting ninth grade 

NSC 

Enrolled in college in 

11th grade 

Dichotomous indicator of whether a student enrolled in college between 
August and July of Year 3 after starting ninth grade 

NSC 

Enrolled in college in 

12th grade 

Dichotomous indicator of whether a student enrolled in college between 
August and July of Year 4 after starting ninth grade 

NSC 

Ever enrolled in 
college in high school 

Dichotomous indicator of whether a student enrolled in college by August 
of Year 4 after starting ninth grade (same as variable, ever enrolled in 
college by Year 4) 

NSC 

Earned any college 
credit in high school 

Dichotomous indicator of whether a student attained any college credit in 
high school 

Survey 

Earned 1 year of 
college credit in high 
school 

Dichotomous indicator of whether a student obtained at least one year of 
college credit in high school 

Survey 

Took at least 1 AP 

course 

Dichotomous indicator of whether a student took at least one AP course in 
high school 

Survey 

Passed at least 1 AP 

exam 

Dichotomous indicator of whether a student passed at least one AP exam 

Survey 

Rigorous Academics 

College preparatory 
math sequence 

Dichotomous indicator of whether a student completed algebra 1, algebra 

II, and geometry in high school 

Survey 

Advanced math course 

Dichotomous indicator of whether a student completed pre-calculus or 
calculus in high school 

Survey 

College preparatory 
science sequence 

Dichotomous indicator of whether a student completed biology, chemistry, 
and physics in high school 

Survey 

Instructional rigor 

A 0-4 scale measuring the frequency with which students reported 
defending their ideas, writing five-page papers, explaining thinking, 
applying what they had learned, engaging in discussions, doing research, 
presenting work, and receiving feedback in 9th through 12th grade (0 = 
never and 4 = almost every day); reliability (a) = 0.82 (scale included eight 
items) 

Survey 

College-going culture 

A 1-4 scale measuring the extent to which the principal, teachers, and 
students expected students in the high school to go to college (1 = 
strongly disagree and 4 = strongly agree); reliability (a) = 0.80 (scale 
included three items) 

Survey 
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Exhibit A.4. High School Experience Measures, Descriptions, and Data Sources (cont.) 



General college Dichotomous variable indicating whether a student reported at least one Survey 

information in school very helpful resource for obtaining general information about college in 
school 


General college Dichotomous variable indicating whether a student reported at least one Survey 

information outside of very helpful resource for obtaining general information about college 
school outside of school 

Instructor support A 1-4 scale measuring the extent to which teachers cared if students Survey 

came to school, praised students for their effort, helped students, listened 
to students, encouraged students, and cared about students (1 = strongly 
disagree and 4 = strongly agree); reliability (a) = 0.88 (scale included five 
items) 


Took the ACT/SAT A dichotomous variable indicating whether a student took the ACT and/or Survey 

SAT in high school 

ACT/SAT score A continuous variable, ranging from 1 to 36, representing students' highest Survey 

ACT or SAT score, with SAT scores converted into ACT units 

Applied for college A dichotomous variable indicating whether a student applied for college Survey 

Applied for financial A dichotomous variable indicating whether a student applied for financial Survey 

aid aid 

ACT/SAT preparation Dichotomous variable indicating whether a student reported at least one Survey 

help in school 3 very helpful resource for preparing for the ACT or SAT in school 

College application Dichotomous variable indicating whether a student reported at least one Survey 

help in school 3 very helpful resource for completing college applications in school 

Financial aid Dichotomous variable indicating whether a student reported at least one Survey 

application help in very helpful resource for completing financial aid forms in school 

school 3 

ACT/SAT preparation Dichotomous variable indicating whether a student reported at least one Survey 

help outside of school 3 very helpful resource for preparing for the ACT or SAT outside of school 

College application Dichotomous variable indicating whether a student reported at least one Survey 

help outside of school 3 very helpful resource for completing college applications outside of school 

Financial aid Dichotomous variable indicating whether a student reported at least one Survey 

application help very helpful resource for completing financial aid forms outside of school 

outside of school 3 


a These items were administered only to students who reported taking the ACT or SAT, filling out college applications, and 
applying for financial aid. 
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Appendix B: Samples and Data Sources 

The study includes two overlapping samples: the administrative data sample and the survey 
sample. The administrative data sample includes all students who participated in a lottery to 
attend 10 Early Colleges between 2005-06 and 2007-08. The survey sample was drawn from 
the administrative data sample. This appendix describes the administrative data sample, 
including the sample size, no-show and crossover rates, and data collection processes. It also 
provides details on the survey sample, including survey weights, response rates, data 
collection processes, and baseline testing. 

Administrative Data 

Sample Size 

The administrative data sample included 1,044 Early College and 1,414 comparison 
students (2,458 students in total). Thirteen students applied to two lotteries, either in 
successive cohort years or at different sites. These students were treated as if they were 
separate (independent) students in determining sample size and in analyses. Two Early 
Colleges had a five-year program, and these sites were included in the study for only the 
2005-06 or 2006-07 cohorts to allow sufficient time for high school graduation. 

Exhibit B.1 shows the sample size for each site and cohort. There were separate lotteries for 
each site and cohort. Three Early Colleges had sublotteries in which lotteries were 
conducted by feeder district, feeder school, or criteria established by the Early College. See 
Appendix D for a description of how sublotteries were treated in analysis. In total, there were 
17 main lotteries in the administrative data sample. The smallest lottery had 39 students and 
the largest lottery had 326 students. 
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Exhibit B.l. Administrative Data Sample Sizes, by Study Site and Cohort 



NOTES'. Shaded cells are years that the school (a) was not open, (b) did not have a lottery, (c) did not have lottery records, or 
(d) did not have a graduating student cohort during the study period because it offered a five-year program. 


No-Show and Crossover Rates 


Exhibit B.2 displays no-show and crossover rates for each lottery in the study. “No-shows” 
are students chosen by lottery to attend an Early College (treatment students) who did not 
enroll in an Early College in the first year of high school. “Crossovers” are students who 
were not selected to attend an Early College (comparison students) who nevertheless 
enrolled in an Early College during the first year of high school. The no-show rate for 
individual lotteries ranged from 0 percent to 58.1 percent, with an overall no-show rate of 
22.0 percent. 54 The overall crossover rate was 1.9 percent. Many lotteries did not have any 
comparison students attending an Early College; however, one lottery had a crossover rate 
of 28.4 percent. 55 For all key outcome variables, we performed sensitivity analyses 
excluding the site with a high no-show rate (Site C) and the site with a high crossover rate 
(Site B) (see Appendix F). 


54 The site with a no-show rate of 58.1 percent, Site C, is located in a district that had independent lotteries for all schools in the 
district. Because students might have entered and won multiple lotteries, many students failed to attend after winning the 
lottery, resulting in a high no-show rate. 

55 The site with a crossover rate of 28.4 percent, Site B, is located near another Early College affiliated with the ECHSI but not 
part of this study. Students in our study who attended this other Early College were coded as receiving the treatment. 
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Exhibit B.2. No-Show and Crossover Rates, by Study Site and Cohort 



Study Cohorts 


2005-06 

2006-07 

2007-08 

Study Sites 

No-shows Crossovers 

No-shows Crossovers 

No-shows Crossovers 



NOTES'. Rates are based on enrollment data for students' first year in high school, n = 2,223. 

Data Collection From Schools, Districts, and State Departments of 
Education 

We collected administrative data covering 2004-05 to 2010-11 from multiple sources, 
including Early Colleges, districts, and state departments of education. For each site, we 
determined the source that could most efficiently provide each type of data. For most sites, 
we collected data directly from the school, district, or state; for four sites, however, we 
worked with a subcontractor who collected the data. When possible, we requested the same 
data from an additional source for corroboration. Data on treatment group status and lottery 
documentation were usually provided by the Early Colleges, and data on student 
background characteristics and outcomes generally came from districts and states. 

We worked closely with every data source provider to understand their data file format and 
variable definitions through phone calls and written documentation, and we obtained the 
sites’ final verification of our interpretation of their data. In particular, we worked closely with 
sites to understand the lottery record data. Of the 10 study sites, 8 provided clear lottery 
documentation, and the lottery data were consistent with the documentation. At two sites, 
Early College staff were not able to provide clear documentation of their lottery processes. 

At one of these sites, school staff partially documented and partially described from memory 
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the decision rules for running the lottery, although the staff members felt confident about 
their recollections. At the second site, school staff had limited recollections of the lottery 
process. 

We have taken two approaches to mitigate any potential bias introduced by the two sites 
that lacked clear descriptions of their lotteries. First, we conducted sensitivity analyses on 
key outcomes with and without each of these sites (see Appendix F for results). Second, we 
examined baseline equivalence and controlled for student baseline characteristics (including 
prior achievement) in all impact analyses. We used two different types of models. In the first 
model, we tested whether treatment group status was predicted by the set of student 
background characteristics (female, minority, low income, first generation, and prior test 
scores), and we found a significant group difference. The model findings for the 
administrative sample are presented in Exhibit B.3. In the second type of model, we tested 
individual characteristics to examine whether each was predicted by treatment group status. 
As noted in Chapter 2 (see Exhibit 2.6), we found a significant difference between the 
groups on ELA prior achievement. 


Exhibit B.3. Likelihood-Ratio Test to Predict Treatment Assignment Using the Administrative Sample 


Model 

-2 Log Likelihood 

Chi-Square 

P-Value 

Empty model 

2,843.2 

... 

... 

Full model 

2,822.3 

20.9 

.0019 


n = 2,458 students (1,044 Early College, 1,414 comparison) 

NOTES: The empty model includes only sublottery indicators as covariates. The full model includes sublotteries and baseline 
characteristics (female, minority, first generation, low income, prior ELA, and prior mathematics). 

Data Collection From the National Student Clearinghouse 

Postsecondary enrollment and degree completion data came from the National Student 
Clearinghouse (NSC). The NSC collects information from more than 3,300 U.S. public and 
private colleges that together enroll more than 96 percent of college students in the nation 
(NSC, 2013). To obtain college enrollment and completion information for students in the 
study, the study team submitted the following information to the NSC: first and last names 
(for all students), birthdates (for 98 percent of students), and middle initials (for 33 percent of 
students). For each student matched to the database, the NSC provided a record of each 
enrollment in a participating postsecondary institution through the fall term of 2012 and 
completion information for certificates, associate’s degrees, and bachelor’s degrees earned 
through the summer of 2012. Each NSC record includes the following: 

■ The name, state, and type of postsecondary institution (i.e., 2- versus 4-year, public 
versus private) in which the student enrolled 
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■ The beginning and ending dates of the student’s enrollment 

■ Whether the student earned a credential, and, if so, the type of credential earned 
(e.g., certificate, associate of arts, bachelor of arts) 

Of the 2,458 students for whom we requested records, the NSC matched 1,691 (69 percent) 
to at least one postsecondary enrollment record. Although the NSC captures the vast 
majority of college enrollments during and after high school, enrollment records could be 
missing if a student attended a non-participating institution, if a student refused to allow his 
or her records to be reported, or if the study did not provide sufficient detail (or incorrect 
information) for a specific match (e.g., if we were unable to provide a birthdate and middle 
initial). Because the NSC data do not allow researchers to distinguish a student who did not 
attend college (or complete a credential) from a student with a record missing for the 
reasons just listed, our findings underestimate college enrollment and completion. 

Survey Data 

Sample Size 

The survey was drawn from the administrative data sample, and includes randomly selected 
students who participated in the 12 2006-07 and 2007-08 lotteries in eight Early Colleges 
that agreed to provide us with student contact information. The size of the survey sample 
was determined by a power analysis conducted during the design phase of this study. At the 
time we conducted the power analysis, we estimated that 15 lotteries would be included in 
the survey. Based on the power analysis, and assuming 15 lotteries, we concluded that 
a total survey sample of 1,500 students with 50 Early College students and 50 comparison 
students per lottery would be sufficient for detecting an effect as small as 7 percentage 
points for both high school graduation and college enrollment (effect sizes = 0.18 and 0.17, 
respectively, assuming that 60 percent of comparison students would graduate from high 
school and 42 percent of comparison students would enroll in college). 

Because we had 12 rather than 15 lotteries in the final survey sample, and there were 
several instances where a site had fewer than 50 students in one or both study conditions 
for a given lottery, we included up to 100 Early College students and 100 comparison 
students in each lottery to reach the target sample size. For lotteries with more than 
100 students in a condition, we randomly sampled 100 students in that condition. 

Overall, 1,416 students were sampled for the survey, representing 58 percent of the total 
administrative sample, 2 of the 3 cohorts, 8 of the 10 sites, and 12 of the 17 lotteries. A few 
students participated in two different lotteries; they were sampled in each lottery as though 
they were independent students. Adjustments were made after initial sampling because 
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some of the students sampled were deemed ineligible for the survey (e.g., deceased, under 
age 18, or not a lottery participant). Students under age 18 at the time of the survey were 
not eligible for the survey because they could not provide their own consent to participate. 
See Exhibit B.4 for the final sample size for each site and lottery after the adjustment. We 
conducted sensitivity testing to compare key outcomes based on administrative data using 
the full administrative data sample and the survey sample, and the comparison showed little 
difference (see Appendix F). 

Exhibit B.4. Final Survey Sample Sizes, by Study Site and Cohort 


Study Sites 

Study Cohorts 

2006-07 

2007-08 

T 

C 

T 

C 

Site A 

27 

43 



Site B 

75 

93 

86 

54 

SiteC 

71 

99 

69 

98 

Site D 

91 

10 

82 

27 

Site E 



21 

18 

Site F 

51 

37 

57 

66 

Site G 



100 

73 

Site H 





Site 1 

40 

28 



Site J 





Total 

355 

310 

415 

336 


We conducted baseline equivalence testing on the survey sample using the same methods 
as for the administrative sample, presented above and in Chapter 2. We used two different 
types of models. In the first model, we tested whether treatment group status was predicted 
by the set of student background characteristics: female, minority, low income, first 
generation, and prior test scores, and we found a significant group difference. The findings 
for the survey sample are presented in Exhibit B.5. In the second type of model, we tested 
individual characteristics to examine whether each was predicted by treatment group status. 
We found no significant difference between the groups for any characteristic. See Exhibit 
B.6 for results of the set of models tested for group equivalence. 
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Exhibit B.5. Likelihood-Ratio Test to Predict Treatment Assignment Using the Survey Sample 


Model 

-2 Log Likelihood 

Chi-Square 

P-Value 

Empty model 

2,102.9 

... 

... 

Full model 

2,090.0 

12.9 

.0448 


n = 1,294 students (724 Early College, 570 comparison) 

NOTES: The empty model includes only sublottery indicators as covariates. The full model includes sublotteries and baseline 
characteristics (female, minority, first generation, low income, prior ELA, and prior mathematics). 


Exhibit B.6. Average Background Characteristics of Early College and Comparison Students and Testing 
for Group Difference on Each Characteristic Using the Survey Sample 


Characteristics 

Early College 

Comparison 

Difference 

Female 

51.2% 

52.0% 

-0.8% 

Minority 

55.7% 

60.2% 

-4.5% 

First generation 

22.2% 

20.4% 

1.8% 

Low income 

47.2% 

43.8% 

3.4% 

Prior achievement in ELA 

.28 

.21 

.07 

Prior achievement in math 

.28 

.29 

-.01 


n = 1,294 students (724 Early College, 570 comparison) 

NOTES'. All data are from the survey data source except for low income and prior test scores, which comes from administrative 
data. Early College group means are unadjusted means, and comparison group means were computed by subtracting the 
estimated group difference from the unadjusted Early College group means. Prior achievement scores were converted to z-scores 
and were standardized using publicly available statewide assessment means and standard deviations. The values in the Difference 
columns may not match the difference between the Early College and Comparison group means due to rounding. 

The p-values for group differences are based on logistic regression for binary student demographic characteristics and OLS 
regression for prior mathematics and ELA scores. 

* p < .05; ** p < .01; *** p < .001 

Pilot Testing and Administration 

We conducted three rounds of pilot testing prior to the launch of the survey. First, we 
conducted a tracking pilot to determine the feasibility of locating students for the survey based 
on contact information from their Early College applications, which were four to five years old 
at the time of the pilot. AIR subcontracted Survey Research Management (SRM), an 
organization that specializes in student tracking and survey administration. We conducted the 
tracking pilot with 160 students in fall 2010, and SRM located 76 percent of the pilot students. 
We thus concluded that conducting a student survey based on retrospective contact 
information was feasible. Second, we piloted the survey instrument by conducting cognitive 
interviews in spring 2011 with 16 students. Third, SRM conducted a timed pilot test in summer 
2011 with 15 students to test the survey length in both online and computer-assisted 
telephone interview (CATI) formats. After the cognitive interviews and timed pilot, we revised 
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the survey to accurately capture information from the students in the study and ensure that the 
survey would take, on average, approximately 30 minutes to complete. 

The survey administration took place from October 2011 to January 2012. The 
subcontractor contacted students primarily using the phone number(s) and address(es) 
provided on their Early College application, regularly calling non-respondents and sending 
mailings throughout the administration timeframe to inform them about the survey and solicit 
their participation. Participants could choose whether to take the survey online or via CATI. 
We provided participating students with an Amazon.com or Target gift card as an incentive. 
Overall, 1,294 students responded to the survey. The response rate was 91.4 percent (94.0 
percent for Early College students, 88.2 percent for comparison students). The cohort- 
specific response rates are presented in Exhibit 2.4 and the participation rates by survey 
mode are presented in Exhibit B.7. 


Exhibit B.7. Percentage of Survey Respondents, by Mode of Administration 



Percentage of Survey Respondents 

Survey Mode 

Overall 

Early College 

Comparison 

CATI 

49.2% 

47.8% 

50.9% 

Online 

50.9% 

52.2% 

49.1% 


Based on the survey sampling rates and response rates, we computed non-response- 
adjusted survey weights as the inverse of the product of sampling rate and response rate 
within each study condition within each lottery. The survey weights were applied to all 
analyses of survey data so that the results can be generalized to all participants in the lotteries 
included in the survey sample. 

Open-Ended Data 

The analytic measures constructed from survey items are described in Appendix A. In 
addition, we included an item at the end of the survey in which students were asked, “Would 
you like to tell us anything else about your high school or college experiences?” We 
received responses from 586 students of the 1,294 survey respondents, including 50.6 
percent of treatment respondents and 38.6 percent of comparison respondents. 

We coded all open-ended responses in a two-stage process, first specifying themes of 
interest based on our conceptual framework (e.g., high school supports) and then using 
open coding to identify additional themes based on student responses (e.g., preparation for 
college). Respondent quotes are used throughout the report to provide context for the 
impact findings. 
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Appendix C: Missing Data 

Exhibits C.1 and C.2 below show the missing data rates for analysis variables prior to 
imputation. 56 Three variables have overall missing rates above 25 percent. The first- 
generation college-going variable originated from survey data but is used in all analyses. 
This variable was missing for all students who did not take the survey, which is the primary 
reason that 49.0 percent of students were missing this information. The high school 
mathematics assessment variable was missing for 33.8 percent of students primarily 
because four sites were located in states with assessments that were not comparable to 
other assessments in the study, and we did not use these scores. “Enrolled in a partner 
institution” was missing for 27.1 percent of students because graduation date was not 
available for all students, which was required to create this variable. 


Exhibit C.1. Pre-imputation Variable Missing Rates for the Administrative Data Sample, Overall and by 
Study Group 



Percentage Missing 

Variable 

Overall 

(n = 2,458) 

Early College 
(n = 1,044) 

Comparison 

(n = 1,414) 

Female 

0.1% 

0.1% 

0.1% 

Minority 

0.4% 

0.6% 

0.2% 

First-generation college-going 3 

49.0% 

33.3% 

60.6% 

Low income 

8.1% 

10.9% 

6.1% 

ELL 

10.9% 

15.4% 

7.6% 

IEP 

24.1% 

29.9% 

19.9% 

Prior achievement in ELA 

16.0% 

14.5% 

17.1% 

Prior achievement in mathematics 

18.1% 

18.4% 

18.0% 

High school achievement in ELA 

12.9% 

8.3% 

16.3% 

High school achievement in mathematics 

33.8% 

35.6% 

32.4% 

Withdrawal from high school 

8.7% 

5.5% 

11.0% 

Drop out 

8.7% 

5.5% 

11.0% 

Local transfer 

8.7% 

5.5% 

11.0% 

High school graduation 

10.2% 

6.5% 

12.9% 


56 As described in Appendix B, it is not possible to determine the rate of missing data on measures from the NSC. Students in 
our sample who lack NSC data are either students who did not attend college, students who attended a college not in the NSC, 
students who refused permission for the NSC to provide their data, or students for whom our name and other information was 
not consistent with their name in the NSC. 
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Exhibit C.l. Pre-imputation Variable Missing Rates for the Administrative Data Sample, Overall and by 
Study Group (cont.) 



Percentage Missing 


Overall 

Early College 

Comparison 

Variable 

(n = 2,458) 

(n = 1,044) 

(n = 1,414) 

Enrolled in a partner institution 11 

27.1% 

17.6% 

34.9% 

Enrolled at any college after high school 
graduation date 

10.2% 

6.5% 

12.9% 

Earned any type of postsecondary degree after 
high school graduation date 

10.2% 

6.5% 

12.9% 


a Survey data served as the source for this variable because it was not consistently available in administrative data. 
b This variable uses the date associated with high school graduation to determine if enrollment in a partner institution is after 
high school; therefore, this variable is missing when we do not have a graduation date for a student. 


Exhibit C.2. Pre-imputation Variable Missing Rates for the Survey Data Sample, Overall and by 
Treatment Status 


Variable 

Percentage Missing 

Overall 

(n = 1,294) 

Early College 

(n = 724) 

Comparison 

(n = 570) 

Female 

0.5% 

0.6% 

0.4% 

Minority 

1.3% 

1.2% 

1.4% 

First generation college-going 

2.7% 

2.9% 

2.5% 

Low income 3 

11.1% 

11.3% 

10.9% 

Prior achievement in ELA 3 

16.9% 

17.4% 

16.1% 

Prior achievement in mathematics 3 

20.1% 

21.8% 

17.9% 

High school GPA 

1.6% 

1.9% 

1.2% 

Developmental course-taking 11 

0.0% 

0.0% 

0.0% 

College GPA b 

2.0% 

1.6% 

2.5% 

Perceptions of academic difficulty in college 11 

0.2% 

0.0% 

0.5% 

Earned any college credit in high school 

2.2% 

2.4% 

2.1% 

Earned one year of college credit in high school 

2.2% 

2.4% 

2.1% 

Took at least one AP course 

6.1% 

9.4% 

1.9% 

Passed at least one AP exam 

6.1% 

9.4% 

1.9% 

College preparatory math sequence 

0.0% 

0.0% 

0.0% 
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Exhibit C.2. Pre-imputation Variable Missing Rates for the Survey Data Sample, Overall and by 
Treatment Status (cont.) 



Percentage Missing 

Variable 

Overall 

(n = 1,294) 

Early College 

(n = 724) 

Comparison 

(n = 570) 

Advanced mathematics course 

0.0% 

0.0% 

0.0% 

College preparatory science sequence 

0.0% 

0.0% 

0.0% 

High school instructional rigor 

0.2% 

0.3% 

0.0% 

High school college-going culture 

0.1% 

0.1% 

0.0% 

General college information in school 

0.0% 

0.0% 

0.0% 

General college information outside of school 

0.0% 

0.0% 

0.0% 

High school instructor support 

0.1% 

0.1% 

0.0% 

Took the ACT/SAT 

0.1% 

0.1% 

0.0% 

ACT/SAT score b 

0.0% 

0.0% 

0.0% 

Applied for college 

0.2% 

0.1% 

0.2% 

Applied for financial aid 

0.2% 

0.1% 

0.2% 

ACT/SAT preparation help in schook 

0.0% 

0.0% 

0.0% 

College application help in schook 

0.0% 

0.0% 

0.0% 

Financial aid application help in schook 

0.0% 

0.0% 

0.0% 

ACT/SAT preparation help outside of schook 

0.0% 

0.0% 

0.0% 

College application help outside of schook 

0.0% 

0.0% 

0.0% 

Financial aid application help outside of schook 

0.0% 

0.0% 

0.0% 


a Administrative data served as the source for this variable because it was unavailable in survey data. 

b The missing rates for these variables are based on the number of eligible students for the item. For example, ACT/SAT items 

were only asked of those who reported taking the ACT or SAT {n = 1,053), questions about help with college applications were 

only asked of those who reported doing each activity (n = 1,067 applied to college, n = 890 applied for financial aid), questions 

about remedial coursework were asked of those who reported enrolling in college after high school (n = 1,002) and college 

academic difficulty and college GPA were asked of those who had completed at least one term of college after high school (n = 

455). 

We handled missing data in impact analyses through multiple imputation using the SAS 
IVEware procedure. The IVEware procedure implements multiple imputation by chained 
equations developed by Raghunathan, Lepkowski, VanHoewyk, and Solenberger (2001). 
The procedure cycles through the variables in the data set, modeling each variable 
conditional on the others, starting with the variable with the fewest missing data. The 
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imputed values are predicted values based on the imputation model, with the appropriate 
random error added to reflect the uncertainty in the predicted values. 

We performed imputation using a master data set that combined data from administrative 
records, the NSC, and the student survey. The imputation model incorporated all variables 
used in the impact analyses, including measures of student background characteristics, high 
school outcomes, college outcomes, and high school experiences, as well as a set of 
treatment-by-student-characteristic interaction terms that were part of the differential impact 
analyses. In addition, the imputation model included filter variables to restrict the imputation of 
missing values for certain variables to students to which the variables applied. For example, 
missing data on survey measures were only imputed for students in the survey sample. 

We generated 10 imputed data sets and conducted all analyses using each of the 10 data 
sets separately. We then combined estimates across the 10 data sets, taking into account 
the uncertainty in imputed values both within and across the imputed data sets. Our primary 
impact analyses were conducted using the fully imputed data sets, including imputed values 
on outcome measures. As a sensitivity test, we also performed each analysis excluding 
students with imputed outcome values. The results from the sensitivity analysis are 
presented in Appendix F. 


American Institutes for Research 


Early College High School Initiative Impact Study 


100 














APPENDIX D: TECHNICAL DETAILS ON IMPACT ANALYSIS 


Appendix D: Technical Details on Impact Analysis 

Multilevel Model to Estimate the Overall Early College 
Impact (Research Question 1) 

Our primary impact analyses are intent-to-treat (ITT) analyses that assess the overall Early 
College impacts on students who were offered admission to an Early College based on a 
lottery. Given the presence of noncompliance (i.e., no-shows and crossovers), we 
supplemented the ITT analyses with analyses of the Early College impacts on lottery 
winners who actually attended an Early College (i.e., compliers). Details about the complier 
effect analyses and the results are presented in Appendix G. 

To estimate the overall ITT effects of Early Colleges across lotteries, we constructed a two- 
level model that takes into account the clustering of students within lotteries. We modeled 
the intercept as a random effect to represent the potential variation in average student 
outcomes across lotteries. We modeled the Early College effect as fixed at the lottery level, 
because the number of lotteries in the study was too small to generate stable estimates of 
the variation in Early College effect across lotteries. Compared with a random-effects model 
with both random intercept and random treatment slope at the lottery level, the fixed-effects 
model is associated with greater statistical power and does not require the assumption that 
the lotteries in the study sample were representative of a larger population of lotteries. 57 One 
limitation of the fixed-effects model, however, is that it does not allow us to generalize study 
findings to Early College admission lotteries beyond those in the study sample, or examine 
the variation in Early College impact across lotteries and factors that may be associated with 
such variation, if any. 

Exhibit D.1 presents the specification of a fixed-effects hierarchical generalized linear model 
(HGLM) for assessing the Early College effect on a binary outcome (graduation from high 
school in this example). The impact model for continuous outcomes (e.g., high school 
achievement scores) was specified similarly based on a hierarchical linear model (HLM). 


57 Student-level non-response adjusted survey weights were applied to analyses of survey-based outcomes; thus the analysis 
results are generalizable to all participants in the lotteries in the survey sample. 
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Exhibit D.l. Hierarchical Generalized Linear Model for Assessing the ITT Effect of Early Colleges on the 
Probability of Graduating From High School 


Level-1 Model (Student Level) 


log[0,,/(l - 0,)] =p 0 j+Pij*EC v +/32j*X ij + JZ = 2 tf 3 m ] *SUBLOTmij) +r ij 

where 

• tp ij is the probability of graduating from high school for student i in lottery j; 

• ECij is a dummy indicator for treatment status: ECj = 1 if the student was assigned to an Early College 

based on lottery and 0 otherwise, centered on lottery mean; 

• X,y is a vector of student characteristics, including gender, race, low income, and standardized achievement 
scores in prior ELAand mathematics, grand-mean centered; 

• SUBLOTmij is a set of effect-coded indicators for the m sublotteries within a lottery with multiple 
sublotteries; 58 

• p 0j is the average graduation rate (in logits) among comparison students in lottery j; 

• Pij is difference in graduation rate between Early College students and comparison students in lottery j; 

• P 2 j is the relationship between student characteristic X and graduation rate in lottery j; 

• P 3 mj is the difference between the average graduation rate for comparison students in sublottery m and the 
average graduation rate for comparison students across all sublotteries in the given lottery with sublotteries; 
and 

• r j is a random error associated with student i in lottery j. 

Level-2 Model (Lottery Level) 

Poj = Too + u 0 j 
Pij = Yio 
p2j = Y20 
P3mj = Y3m0 

where 

• yoo is the average graduation rate (in logits) among comparison students across all lotteries; 

• yio is the average difference in graduation rate between Early College students and comparison students 
across all lotteries; 

• Y 20 is the average relationship between student characteristic X and graduation rate across all lotteries; and 

• Y 3 mo is the difference between the average graduation rate for comparison students in sublottery m and the 
average graduation rate for comparison students across all sublotteries in the given lottery with sublotteries. 

• u q is a random error associated with lottery j, representing variation across lotteries in the average 
graduation rate for comparison students. 


The model was estimated using the adaptive Gaussian-Hermite quadrature (AGQ) method, 
which is generally considered the preferred estimation method for multilevel logit models. It 


58 For a given lottery with m sublotteries, SUBLOTmij was coded -1 for students in the omitted reference sublottery (i.e., if m = 
1), 1 for students in sublottery m within the given lottery, and 0 for all other students. There is one set of sublottery indicators 
for each lottery with sublotteries in the level-1 equation, although we show only one set here for simplicity. 
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tends to produce more accurate results than alternative estimation methods such as the 
penalized quasi-likelihood method (i.e., the default of the HGLM module in the HLM7.0 
program) or the pseudo-likelihood method (i.e., the default of the SAS GLIMMIX procedure), 
particularly when the higher-level variance is substantial. The estimate of primary interest 
from the model is y 10 , which represents a precision weighted overall treatment effect across 
all lotteries in the study sample, with larger weights for lotteries with more students. The 
analysis was repeated using each of the 10 multiply imputed data sets, and the estimates 
were combined across data sets, taking into account the uncertainty in imputed values both 
within and across the imputed data sets. 

Multilevel Model to Estimate the Variation in Early College 
Impact Across Different Types of Students (Research 
Question 2) 

In addition to the overall Early College effects on student outcomes, we also examined the 
potential differential effects of Early Colleges on students with different background 
characteristics by incorporating treatment-by-student-characteristic interactions into the 
impact model. For example, we measured the difference in Early College effect on high 
school graduation between minority and non-minority students by adding a treatment-by- 
non-minority interaction term to the main impact model (see Exhibit D.2). The lottery-level 
estimate for the interaction term (y 40 ) captures the average difference in Early College effect 
on high school graduation between minority and non-minority students across all lotteries in 
the study sample. 
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Exhibit D.2. Hierarchical Generalized Linear Model for Assessing the Difference Between Minority and 
Non-minority Students in Early College Effect on the Probability of Graduating From High School 


Level-1 Model (Student Level) 


log[0,/(l - m =f]oi + fhj*EC i j+/hj*Xi i + £™ =2 (/?3 mj * SUBLOTmij ) + 
fi 4 j* EQj miNORITYy + Vij 

where 

• ECij *MINORITYij is the interaction between treatment status (EC) and minority status for student i in lottery j; 

• p^j is the difference in Early College effect between non-minority and minority students (i.e., differential Early 

College effect) in lottery j; and 

• Other terms are defined similarly to those in the main impact model shown in Appendix D. 



We performed similar analyses to test the potential differential effects of Early Colleges 
associated with gender, low-income status, first-generation status, and prior achievement. 
The model presented in Exhibit D.2 requires that, within each lottery, both Early College 
students and comparison students be present within each of the subgroups we examine. 
Lotteries that do not satisfy this requirement must be removed from analyses because, to 
estimate the impact of attending an Early College within a subgroup, there must be both 
Early College students and comparison students within the subgroup. For example, within 
two lotteries in the study, all Early College students were minority, and thus Early College 
impacts among non-minority students could not be estimated. To estimate the differential 
Early College impact for minority and non-minority students, we therefore removed these 
two lotteries from our analyses. 
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Appendix E: Summary of Impact Findings 

This appendix provides detailed ITT estimates for all outcomes examined in the overall 
impact analyses and differential impact analyses. Exhibits E.1- E.4 summarize the results 
from the overall impact analyses for binary and continuous variables, and Exhibits E.5 and 
E.6 summarizes the results from the analyses of differential Early College impact on three 
key outcomes (graduation from high school, ever enrolled in college, and ever earned a 
postsecondary degree). The tables include the percentage of Early College (EC) and 
comparison (C) students who experienced the outcomes as well as effect sizes and p- 
values. 59 


Exhibit E.l. Summary of ITT Estimates of the Overall Early College Impact on High School and College 
Binary Outcomes 


Outcome 

Analysis 

n 

Effect 

in 

Logits 

Odds 

Ratio 

Std. 

Error 

(logit) 

Probabilities 

EC C 

Effect 

Size 

P-Value 

Withdrawal from initial high school 

2,458 

-0.21 

0.81 

0.11 

32.2% 

36.9% 

-0.127 

.0552 

Drop out 

2,458 

0.02 

1.02 

0.24 

4.9% 

4.8% 

0.011 

.9408 

Local transfer 

2,458 

-0.15 

0.86 

0.13 

16.6% 

18.7% 

-0.089 

.2733 

High school graduation 

2,458 

0.39 

1.48 

0.15 

86.0% 

80.6% 

0.236 

.0103 

Ever enrolled in college 

2,458 

0.50 

1.65 

0.11 

79.7% 

70.5% 

0.302 

<.0001 

Enrolled in college by Year 4 

2,458 

1.75 

5.77 

0.11 

63.1% 

22.9% 

1.062 

<.0001 

Enrolled in college by Year 5 

2,458 

0.54 

1.71 

0.11 

77.4% 

66.7% 

0.326 

<.0001 

Enrolled by Year 4 for 2005-06 and 
2006-07 cohorts 

1,458 

1.73 

5.62 

0.15 

61.0% 

21.8% 

1.046 

<.0001 

Enrolled by Year 5 for 2005-06 and 
2006-07 cohorts 

1,458 

0.49 

1.64 

0.14 

74.8% 

64.5% 

0.299 

.0005 

Enrolled by Year 6 for 2005-06 and 

1,458 

0.49 

1.64 

0.14 

77.7% 

67.9% 

0.300 

.0006 


2006-07 cohorts 


59 Effect sizes for binary variables are based on the following formula (Hedges, 1981): 


x,-x, 

'(/I,-1)57+(H,-1)5? 

(n,+n 2 - 2 ) 
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Exhibit E.1. Summary of ITT Estimates of the Overall Early College Impact on High School and College 
Binary Outcomes (cont.) 


Outcome 

Analysis 

n 

Effect 

in 

Logits 

Odds 

Ratio 

Std. 

Error 

(logit) 

Probabilities 

EC C 

Effect 

Size 

P-Value 

Ever enrolled in a 2-year college 

2,458 

0.87 

2.39 

0.11 

59.2% 

37.8% 

0.528 

<.0001 

Enrolled in a 2-year college by 

Year 4 

2,458 

1.95 

7.05 

0.13 

48.0% 

11.6% 

1.184 

<.0001 

Enrolled in a 2-year college by 

Year 5 

2,458 

1.02 

2.77 

0.11 

55.7% 

31.1% 

0.618 

<.0001 

Ever enrolled in a 4-year college 

2,458 

0.26 

1.30 

0.10 

53.5% 

47.1% 

0.157 

.0131 

Enrolled in a 4-year college by 

Year 4 

2,458 

1.20 

3.32 

0.17 

23.6% 

8.5% 

0.727 

<.0001 

Enrolled in a 4-year college by 

Year 5 

2,458 

0.23 

1.26 

0.11 

48.9% 

43.2% 

0.138 

.0338 

Enrolled in partner institutions 

1,942 

0.49 

1.63 

0.11 

55.8% 

43.7% 

0.300 

<.0001 

Any postsecondary degree 

2,458 

2.69 

14.79 

0.23 

22.2% 

1.9% 

1.633 

<.0001 

Ever earned a certificate 

2,458 

0.54 

1.71 

0.48 

1.2% 

0.7% 

0.327 

.2658 

Ever earned an A.A. degree 

2,458 

2.92 

18.56 

0.26 

21.0% 

1.4% 

1.770 

<.0001 

Ever earned a B.A. degree 

2,458 

3.99 

54.31 

1.11 

2.3% 

0.0% 

2.421 

.0003 

Postsecondary degree earned by 

Year 4 

2,458 

3.61 

36.94 

0.35 

19.8% 

0.7% 

2.188 

<.0001 

Postsecondary degree earned by 

Year 5 

2,458 

3.13 

22.87 

0.28 

21.4% 

1.2% 

1.897 

<.0001 

Postsecondary degree earned by 

Year 4 for 2005-06 and 2006-07 
cohorts 

1,458 

4.44 

85.06 

0.68 

17.6% 

0.3% 

2.693 

<.0001 

Postsecondary degree earned by 

Year 5 for 2005-06 and 2006-07 
cohorts 

1,458 

3.21 

24.73 

0.39 

19.3% 

1.0% 

1.944 

<.0001 

Postsecondary degree earned by 

Year 6 for 2005-06 and 2006-07 
cohorts 

1,458 

2.66 

14.27 

0.31 

20.0% 

1.7% 

1.611 

<.0001 

Enrolled in any college between 

Year 5 and Year 7 

2,458 

0.13 

1.14 

0.10 

71.1% 

68.4% 

.078 

.2181 
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APPENDIX E: SUMMARY OF IMPACT FINDINGS 


Exhibit E.1. Summary of ITT Estimates of the Overall Early College Impact on High School and College 
Binary Outcomes (cont.) 




Effect 


Std. 

Probabilities 



Outcome 

Analysis 

n 

in 

Logits 

Odds 

Ratio 

Error 

(logit) 

EC 

C 

Effect 

Size 

P-Value 

Enrolled at any college after high 
school graduation date 

1,791 

0.07 

1.07 

0.14 

81.3% 

80.2% 

0.042 

.6138 

Enrolled in college between Year 5 
and Year 7 at 2-year college 

2,458 

-0.09 

0.91 

0.10 

32.8% 

34.8% 

-0.057 

.3591 

Enrolled in college between Year 5 
and Year 7 at 4-year college 

2,458 

0.23 

1.26 

0.10 

51.2% 

45.5% 

0.138 

.0270 

Earned any type of postsecondary 
degree between Year 5 and Year 7 

2,458 

1.15 

3.17 

0.31 

4.5% 

1.5% 

0.700 

.0002 

Earned any type of postsecondary 
degree after high school graduation 
date 

1,791 

1.28 

3.60 

0.26 

10.8% 

3.3% 

0.776 

<.0001 

Earned A.A. degree between Year 5 
and Year 7 

2,458 

1.02 

2.78 

0.42 

2.0% 

0.7% 

0.620 

.0144 

Earned B.A. degree between Year 5 
and Year 7 

2,458 

3.99 

54.31 

1.11 

2.3% 

0.0% 

2.421 

.0003 

Developmental coursework 

1,002 

-0.24 

0.79 

0.17 

18.2% 

22.0% 

-0.146 

.1472 

Persistence in college 

1,569 

0.05 

1.05 

0.13 

67.4% 

66.4% 

0.028 

.7289 


NOTES: The treatment group probabilities are unadjusted probabilities; the comparison group probabilities were computed 
based on the unadjusted treatment group probabilities and estimated Early College effects. 
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APPENDIX E: SUMMARY OF IMPACT FINDINGS 


Exhibit E.2. Summary of ITT Estimates of the Overall Early College Impact on High School Experiences, 
Binary Outcomes 


Outcome 


Analysis Effect in 
n Logits 


Std. Error 
(logit) 


Probabilities 



Enrolled in college in 9th grade 

2,458 

6.05 

422.11 

0.53 

28.9% 

4 

3.664 

<.0001 

Enrolled in college in 10th grade 

2,458 

3.45 

31.64 

0.19 

47.4% 

2.8% 

2.094 

<.0001 

Enrolled in college in 11th grade 

2,458 

2.45 

11.63 

0.15 

46.2% 

6.9% 

1.487 

<.0001 

Enrolled in college in 12th grade 

2,458 

1.52 

4.58 

0.11 

52.5% 

19.4% 

0.923 

<.0001 

Ever enrolled in college in high school 

2,458 

1.75 

5.77 

0.11 

63.1% 

22.9% 

1.062 

<.0001 

Earned any college credit in high 
school 

1,294 

1.99 

7.32 

0.15 

66.3% 

21.2% 

1.207 

<.0001 

Earned 1 year of college credit in high 
school 

1,294 

2.81 

16.69 

0.19 

49.1% 

5.5% 

1.706 

<.0001 

Took at least 1 AP course 

1,294 

-1.50 

0.22 

0.15 

21.8% 

55.5% 

-0.910 

<.0001 

Passed at least 1 AP exam 

1,294 

-1.68 

0.19 

0.21 

8.5% 

33.3% 

-1.017 

<.0001 

College preparatory math sequence 

1,294 

-0.15 

0.86 

0.13 

75.4% 

78.1% 

-0.091 

.2421 

Advanced math course 

1,294 

-0.07 

0.93 

0.13 

58.2% 

59.9% 

-0.043 

.5869 

College preparatory science sequence 

1,294 

0.35 

1.43 

0.12 

50.7% 

41.9% 

0.215 

.0035 

General college information in school 

1,294 

0.27 

1.31 

0.14 

83.6% 

79.5% 

0.165 

.0602 

General college information outside of 
school 

1,294 

-0.05 

0.95 

0.11 

38.8% 

40.0% 

-0.031 

.6395 

Took the ACT/SAT 

1,294 

0.00 

0.99 

0.14 

82.4% 

82.6% 

-0.006 

.9481 

Applied for college 

1,294 

0.02 

1.02 

0.15 

83.1% 

82.9% 

0.010 

.9104 

Applied for financial aid 

1,294 

0.15 

1.16 

0.12 

70.8% 

67.7% 

0.088 

.2269 

ACT/SAT preparation help in school 

1,053 

0.20 

1.22 

0.13 

54.2% 

49.2% 

0.122 

.1109 

College application help in school 

1,067 

0.43 

1.54 

0.14 

75.0% 

66.1% 

0.263 

.0015 

Financial aid application help in school 

890 

0.32 

1.38 

0.14 

63.1% 

55.3% 

0.195 

.0233 

ACT/SAT preparation help outside of 
school 

1,053 

-0.06 

0.94 

0.15 

19.3% 

20.2% 

-0.036 

.6913 

College application help outside of 
school 

1,067 

-0.07 

0.93 

0.13 

31.5% 

33.1% 

-0.045 

.5558 

Financial aid application help outside of 
school 

890 

0.25 

1.28 

0.13 

47.1% 

41.0% 

0.151 

.0640 


NOTES: The treatment group probabilities are unadjusted probabilities; the comparison group probabilities were computed 
based on the unadjusted treatment group probabilities and estimated Early College effects. 

$ Too few students to report 
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APPENDIX E: SUMMARY OF IMPACT FINDINGS 


Exhibit E.3. Summary of ITT Estimates of the Overall Early College Impact on High School and College 
Continuous Outcomes 


Outcome 

Analysis 

n 

Effect 

Std. 

Error 

Mean 

EC C 

Effect 

Size 

P-Value 

Achievement in ELA 

2,458 

0.14 

0.04 

0.37 

0.23 

0.147 

.0001 

Achievement in mathematics 

2,458 

0.05 

0.04 

0.28 

0.23 

0.056 

.1908 

High school GPA 

1,294 

0.00 

0.04 

2.98 

2.98 

-0.004 

.9440 

College GPA 

455 

-0.02 

0.07 

3.07 

3.09 

-0.024 

.8150 

Perceptions of academic difficulty 
in college 

455 

-0.13 

0.06 

2.04 

2.17 

-0.220 

.0322 


NOTES: The treatment group means are unadjusted means; the comparison group means were computed based on the 
unadjusted treatment group means and estimated Early College effects. 


Exhibit E.4. Summary of ITT Estimates of the Overall Early College Impact on High School Experiences 
Continuous Outcomes 





Std. 

Error 

Mean 

Effect 

Size 


Outcome 

Analysis 

n 

Effect 

EC 

C 

P-Value 

Instructional rigor 

1,294 

0.22 

0.04 

2.7 

2.5 

0.346 

<.0001 

College-going culture 

1,294 

0.18 

0.03 

3.5 

3.3 

0.318 

<.0001 

Instructor support 

1,294 

0.18 

0.03 

3.4 

3.2 

0.314 

<.0001 

ACT/SAT score 

1,053 

.36 

.30 

22.7 

22.4 

.068 

.2397 


NOTES'. The treatment group means are unadjusted means; the comparison group means were computed by subtracting the 
estimated Early College effects from the unadjusted treatment group means. ACT/SAT score is reported in ACT units. 
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APPENDIX E: SUMMARY OF IMPACT FINDINGS 


Exhibit E.5. Differential Early College Impact on High School Graduation, Ever Enrolled in College, and 
Any Postsecondary Degree, by Student Background Characteristics 



X = 1 

(Probability) 

X = 0 

(Probability) 

Differential Effect 

Student 

Characteristics 

EC 

C 

Difference 

N 

EC 

C 

Difference 

N 

Odds 

Ratio 

Difference 
in Effect 

P-Value 

High School Graduation 

Female 

84.9% 

83.1% 

1.8% 

1,264 

87.1% 

77.8% 

9.4% 

1,194 

0.6 

-7.5% 

.0764 

Minority 

87.4% 

81.8% 

5.6% 

965 

89.0% 

82.5% 

6.5% 

855 

0.9 

-0.9% 

.7523 

Low income 

82.6% 

73.7% 

8.9% 

1,263 

88.8% 

86.6% 

2.1% 

1,125 

1.4 

6.7% 

.2759 

First generation 

80.0% 

77.8% 

2.3% 

956 

88.6% 

81.7% 

6.9% 

1,502 

0.7 

-4.6% 

.1984 

Ever Enrolled in College 

Female 

81.0% 

74.9% 

6.1% 

1,264 

78.3% 

65.6% 

12.7% 

1,194 

0.8 

-6.7% 

.2018 

Minority 

80.3% 

72.2% 

8.1% 

965 

83.2% 

72.9% 

10.3% 

855 

0.9 

-2.2% 

.5399 

Low income 

74.9% 

64.3% 

10.6% 

1,263 

84.5% 

75.5% 

9.1% 

1,125 

0.9 

1.5% 

.7801 

First generation 

73.9% 

68.3% 

5.5% 

956 

82.2% 

71.5% 

10.7% 

1,502 

0.7 

-5.2% 

.3091 

Any Postsecondary Degree 

Female 

22.8% 

1.2% 

21.6% 

1,264 

21.7% 

2.9% 

18.8% 

1,194 

2.6 

2.8% 

.0310 

Minority 

26.5% 

.9% 

25.6% 

965 

22.8% 

2.8% 

19.9% 

855 

3.9 

5.7% 

.0081 

Low income 

19.7% 

.8% 

18.9% 

1,263 

24.5% 

3.4% 

21.2% 

1,125 

3.4 

-2.3% 

.0177 

First generation 

21.1% 

1.4% 

19.7% 

956 

22.6% 

2.1% 

20.5% 

1,502 

1.4 

-0.7% 

.5537 


NOTES: X = 1 for female, minority, low income, first generation; X = 0 for male, non-minority, not low income, not first 
generation. The treatment group probabilities within a given student subgroup are unadjusted probabilities; the comparison 
group probabilities were computed based on the unadjusted treatment group probabilities and estimated Early College effects 
within the subgroup. The values in the Difference columns may not match the difference between the Early College and 
Comparison group means due to rounding. To analyze differential effects for minority and non-minority students, two lotteries 
were removed from analyses because zero Early College students were non-minority, and thus Early College impacts among 
non-minority students could not be estimated within these lotteries. Similarly, one lottery was removed from analyses of 
differential Early College effects among low-income students because zero students within this lottery were low-income. 
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APPENDIX E: SUMMARY OF IMPACT FINDINGS 


Exhibit E.6. Differential Early College Impact on High School Graduation, Ever Enrolled in College, and 
Any Postsecondary Degree, by Prior Achievement 



X = -1 Standard Deviation 

(Probability) 

x = os 

tandard Deviation 

Probability) 

X 

= +1 Standard 
Deviation 

Probability) 

Differential Effect 

Prior 

Achievement 

EC 

C 

Difference 

EC 

C 

Difference 

EC 

C 

Difference 

Odds 

Ratio 

N 

P-Value 

High School Graduation 

ELA 

81.3% 

77.9% 

3.3% 

87.7% 

81.7% 

6.0% 

92.1% 

84.9% 

7.1% 

1.3 

2,458 

.1148 

Mathematics 

81.0% 

77.3% 

3.8% 

87.7% 

81.8% 

5.9% 

92.2% 

85.6% 

6.7% 

1.3 

2,458 

.1536 

Ever Enrolled in College 

ELA 

73.6% 

65.5% 

8.1% 

80.0% 

70.5% 

9.5% 

85.1% 

75.0% 

10.2% 

1.1 

2,458 

.3110 

Mathematics 

72.4% 

60.9% 

11.5% 

79.8% 

70.6% 

9.1% 

85.5% 

78.8% 

6.8% 

1.0 

2,458 

.8194 

Any Postsecondary Degree 

ELA 

7.6% 

1.2% 

6.4% 

8.7% 

0.8% 

7.9% 

10.0% 

0.5% 

9.5% 

1.8 

2,458 

.0308 

Mathematics 

4.8% 

0.9% 

3.9% 

8.7% 

0.9% 

7.9% 

15.4% 

0.8% 

14.5% 

2.0 

2,458 

.0143 


NOTES: The treatment group probabilities for a given level of prior achievement (i.e., 1 SD below the state average, state 
average, and 1 SD above the state average) are predicted probabilities when all control variables other than the prior 
achievement measure were set to their grand means. The comparison group probabilities were computed based on the 
predicted treatment group probabilities and the estimated differential Early College effects. The values in the Difference 
columns may not match the difference between the Early College and Comparison group means due to rounding. 
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APPENDIX F: SENSITIVITY ANALYSES 


Appendix F: Sensitivity Analyses 

We performed five types of sensitivity analyses to assess the overall Early College impact 
on primary outcomes (high school graduation, college enrollment, and any postsecondary 
degree). First, we conducted impact analyses that excluded particular sites due to: a high 
no-show rate, a high crossover rate, incomplete documentation of the lottery, unavailability 
of high school mathematics assessment scores, or significant group difference in prior ELA 
assessment scores. Second, we performed impact analyses of outcomes from the 
administrative data using the survey sample and compared the results to those based on 
the administrative data sample. Third, we conducted impact analyses on selected outcomes 
incorporating random variation in the treatment effect across sites and compared the results 
with those based on the fixed-effects model. Fourth, we conducted the impact analyses for 
primary outcomes excluding students with missing outcome data, and compared the results 
with the full-sample results based on imputed data. Finally, we compared results for selected 
outcomes using ITT models that excluded baseline characteristics as covariates with results 
from our main model. 

Comparison of Impact Results With and Without Specific 
Sites 

Exhibit F.1 compares findings for three outcomes using the main impact model with findings 
from analyses that excluded specific sites. We tested the sensitivity of the impact findings to 
the exclusion of Sites A and F because their lotteries were not fully documented. Site A’s 
staff described the lottery process with reasonable detail. However, the data did not fully 
match their explanation of who was included in the lottery. Site F’s lottery was also not well 
documented. 

We also conducted sensitivity analyses that excluded Site B, because it had a high 
crossover rate (28.4 percent in 2006-07), and its lottery winners had high ELA scores at 
baseline. Some students who lost the lottery attended another Early College that was in the 
same district, and we coded enrollees of this other Early College as receiving the treatment, 
thus counting as crossovers. 

In another sensitivity analysis, we excluded Site C because it had high no-show rates in all 
three lotteries (45.5 percent in 2005-06, 58.1 percent in 2006-07, and 57.4 percent in 
2007-08). This site is situated in a district that ran district-wide lotteries for all schools. Each 
school’s lottery was independent, so students might have applied to and won multiple 
lotteries. As a result, about half of students who won the Site C lottery did not attend, 
presumably because they also won another lottery at a school they preferred to attend. 
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APPENDIX F: SENSITIVITY ANALYSES 


Exhibit F.l. Results of Early College Impact on High School Graduation, College Enrollment, and Degree 
Attainment With and Without Specific Sites 



Analysis 

Effect in 

Odds 

Std. 

Probabilities 

l 

Effect 


Analysis Sample 

n 

Logits 

Ratio 

Error 

EC 

C 

Size 

P-Value 

High School Graduation 

Full sample 

2,458 

0.39 

1.48 

0.15 

86.0% 

80.6% 

0.236 

.0103 

Excluding Site A 

2,388 

0.40 

1.49 

0.15 

85.9% 

80.3% 

0.241 

.0094 

Excluding Site B 

2,129 

0.26 

1.29 

0.16 

85.0% 

81.5% 

0.156 

.1125 

Excluding Site C 

1,518 

0.74 

2.09 

0.22 

90.3% 

81.7% 

0.447 

.0008 

Excluding Site F 

2,247 

0.34 

1.40 

0.15 

84.5% 

79.5% 

0.206 

.0220 

Ever Enrolled in College 

Full sample 

2,458 

0.50 

1.65 

0.11 

79.7% 

70.5% 

0.302 

<.0001 

Excluding Site A 

2,388 

0.53 

1.70 

0.11 

79.9% 

70.0% 

0.323 

<.0001 

Excluding Site B 

2,129 

0.44 

1.55 

0.12 

78.8% 

70.6% 

0.265 

.0004 

Excluding Site C 

1,518 

0.72 

2.05 

0.15 

84.6% 

72.8% 

0.435 

<.0001 

Excluding Site F 

2,247 

0.45 

1.56 

0.12 

78.0% 

69.4% 

0.271 

<.0001 

Any Postsecondary Degree 

Full sample 

2,458 

2.69 

14.79 

0.23 

22.2% 

1.9% 

1.633 

<.0001 

Excluding Site A 

2,388 

2.64 

14.02 

0.23 

22.2% 

2.0% 

1.600 

<.0001 

Excluding Site B 

2,129 

2.83 

17.01 

0.24 

25.7% 

2.0% 

1.718 

<.0001 

Excluding Site C 

1,518 

3.02 

20.58 

0.26 

26.4% 

1.7% 

1.833 

<.0001 

Excluding Site F 

2,247 

2.08 

7.98 

0.24 

16.7% 

2.4% 

1.259 

<.0001 


NOTES: The treatment group probabilities are unadjusted probabilities; the comparison group probabilities were computed 
based on the unadjusted treatment group probabilities and estimated Early College effects. 


Finally, we tested the sensitivity of findings about Early College impact on high school 
mathematics achievement to the exclusion of four sites (Sites G, H, I, and J) because they did not 
have high school mathematics scores due to special circumstances related to their states’ testing 
programs (see Exhibit F.2). 

In general, the results of the analyses excluding these sites were very similar to the results 
based on the full sample. There was only one difference in the statistical significance of the 
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APPENDIX F: SENSITIVITY ANALYSES 


impact estimates: When Site B was excluded from the analyses for high school graduation, 
there was no longer a significant difference between Early College and comparison students. 


Exhibit F.2. Results of Early College Impact on High School Mathematics Assessments With and Without 
Specific Sites 


Analysis Sample 

Analysis 

n 

Effect in 
Logits 

Odds 

Ratio 

Std. 

Error 

Means 

EC C 

Effect 

Size 

P-Value 

Full sample 

2,458 

0.05 

NA 

0.04 

0.28 

0.23 

0.056 

.1908 

Excluding Sites G, H, 1, 

and J 

1,917 

0.04 

NA 

0.04 

0.20 

0.16 

0.043 

.2550 


NOTES: The treatment group means are unadjusted means; the comparison group means were computed based on the 
unadjusted treatment group means and estimated Early College effects. 


Comparison of Impact Results Based on the 
Administrative Data Sample and Survey Sample 

In the second set of sensitivity analyses, we examined whether the impact findings based on 
the survey sample were similar to the findings based on the administrative data sample. As 
shown in Exhibit F.3, the findings for the three outcomes examined were similar based on 
the two samples. 


Exhibit F.3. Results of Early College Impact on High School Graduation, College Enrollment, and Degree 
Attainment Based on the Administrative Data Sample and Survey Sample 


Analysis 

Analysis 

Effect in 

Odds 

Std. 

Probabilities 

Effect 


Sample 

n 

Logits 

Ratio 

Error 

EC 

C 

Size 

P-Value 

High School Graduation 

Full sample 

2,458 

0.39 

1.48 

0.15 

86.0% 

80.6% 

0.236 

.0103 

Survey 

1,286 

0.41 

1.51 

0.19 

87.2% 

81.9% 

0.248 

.0339 

Ever Enrolled in College 

Full sample 

2,458 

0.50 

1.65 

0.11 

79.7% 

70.5% 

0.302 

<.0001 

Survey 

1,286 

0.42 

1.51 

0.15 

78.9% 

71.2% 

0.252 

.0052 

Any Postsecondary Degree 

Full sample 

2,458 

2.69 

14.79 

0.23 

22.2% 

1.9% 

1.633 

<.0001 

Survey 

1,286 

2.84 

17.05 

0.31 

19.5% 

1.4% 

1.719 

<.0001 


NOTES: The treatment group probabilities are unadjusted probabilities; the comparison group probabilities were computed 
based on the unadjusted treatment group probabilities and estimated Early College effects. 
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APPENDIX F: SENSITIVITY ANALYSES 


Comparison of Impact Results Based on the Fixed-Effects 
Model and a Random-Effects Model 

Our main impact analyses were based on a fixed-effects model that estimated a common 
Early College impact across lotteries because the number of lotteries was too small to 
provide stable estimates of the variation in Early College impact across lotteries for some 
outcomes. As a sensitivity analysis, we reran the impact analysis using a random-effects 
model for two outcomes that did not have estimation problems when the Early College effect 
was modeled as a random effect at the lottery level. 

As shown in Exhibit F.4, the findings about the Early College impact on college enrollment 
and postsecondary degree attainment based on the fixed-effects model and the random- 
effects model were similar. Exhibit F.5 shows that there was no significant variation in the 
treatment effect on these two outcomes across lotteries, and Exhibits F.6 and F.7 present bar 
charts that depict the variation in Early College impact across lotteries. 


Exhibit F.4. Comparison of Impact Results on the Fixed-Effects Model and Random-Effects Model for Ever 
Enrolling in College and Ever Earning a Postsecondary Credential Based on a Random-Effects Model 


Outcome 

Analysis 

Sample 

Effect in 
Logits 

Odds 

Ratio 

Std. 

Error 

Probabilities 

EC C 

Effect 

Size 

P-Value 

Ever enrolled in 
college 

Fixed 

0.50 

1.65 

0.11 

79.7% 

70.5% 

0.302 

<.0001 

Random 

0.61 

1.84 

0.17 

79.7% 

68.1% 

0.369 

.0003 

Any postsecondary 
degree 

Fixed 

2.69 

14.79 

0.23 

22.2% 

1.9% 

1.633 

<.0001 

Random 

1.95 

7.00 

0.61 

22.2% 

3.9% 

1.179 

.0015 


NOTES: The analysis sample included 2,458 students (1,044 treatment, 1,414 comparison) and 17 lotteries. The treatment 
group probabilities are unadjusted probabilities; the comparison group probabilities were computed based on the unadjusted 
treatment group probabilities and estimated Early College effects. 


Exhibit F.5. Variance of the Early College Impact on Ever Enrolling in College and Ever Earning a 
Postsecondary Credential Based on a Random-Effects Model 



Variance of 
EC Effect 




Outcome 

(logit 2 ) 

Std. Error 

T-Statistic 

P-Value 

Ever enrolled in college 

0.18 

0.13 

1.38 

.1677 

Any postsecondary degree 

3.62 

2.09 

1.73 

.0838 


NOTES: The analysis sample included 2,458 students (1,044 treatment, 1,414 comparison) and 17 lotteries. The treatment 
group probabilities are unadjusted probabilities; the comparison group probabilities were computed based on the unadjusted 
treatment group probabilities and estimated Early College effects. 
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APPENDIX F: SENSITIVITY ANALYSES 


Exhibit F.6. Lottery-Specific Early College Impact on Ever Enrolled in College Based on a Random- 
Effects Model 



A -1 B-1 B-2C-1 C-2C-3D-1 D-2E-1 E-2 F-1 F-2G-1H-1 1-1 1-2 J-1 


Lottery 


Exhibit F.7. Lottery-Specific Early College Impact on Any College Degree Based on a Random- 
Effects Model 



A- 1 B-1 B-2C-1 C-2C-3D-1 D-2E-1 E-2 F-1 F-2 G-1 H-1 1-1 1-2 J-1 


Lottery 


NOTES: The large odds ratio for lottery F-2 is due to the fact that no comparison students in this lottery obtained 
postsecondary degrees and Early College students demonstrated very high rates of degree completion. 


American Institutes for Research 


Early College High School Initiative Impact Study 


116 

























APPENDIX F: SENSITIVITY ANALYSES 


Comparison of Impact Results That Included and Excluded 
Imputed Outcome Data 

All of the analyses presented in the body of the report were based on fully imputed data sets 
with imputed values for both outcomes and covariates. As a sensitivity analysis, we reran 
our main ITT model for three outcomes excluding students with missing outcome data. 
Because variables created from NSC data (i.e., college enrollment and college degree) do 
not have missing cases, we instead used high school graduation, ELA assessment score, 
and earned college credit during high school for this set of analyses. Exhibits F.8 and F.9 
show that the Early College impact on high school graduation became non-significant after 
students with missing outcome data were excluded from the analysis. Findings for the other 
two outcomes, however, were similar with and without imputed outcome data. 


Exhibit F.8. Results of Impact Analyses That Included and Excluded Missing Outcome Data on High 
School Graduation, and Earned College Credit During High School 


Outcome 

Analysis 

Sample 

Analysis 

n 

Effect 

in 

Logits 

Odds 

Ratio 

Std. 

Error 

Probabilities 

EC C 

Effect 

Size 

P-Value 

High school 

Full sample 

2,458 

0.39 

1.48 

0.15 

86.0% 

80.6% 

0.236 

.0103 

graduation 

Excluding 
missing outcome 
data 

2,207 

0.17 

1.19 

0.15 

88.0% 

86.1% 

0.105 

.2579 

Earned 
college credit 
during high 
school 

Full sample 

1,294 

1.99 

7.32 

0.15 

66.3% 

21.2% 

1.207 

<.0001 

Excluding 
missing outcome 
data 

1,265 

2.08 

8.04 

0.15 

66.6% 

19.9% 

1.264 

<.0001 


NOTES: The treatment group probabilities are unadjusted probabilities; the comparison group probabilities were computed 
based on the unadjusted treatment group probabilities and estimated Early College effects. 


Exhibit F.9. Results of Impact Analyses That Included and Excluded Missing Outcome Data on ELA 
Assessment Score 


Outcome 

Analysis 

Sample 

Analysis 

n 

Effect 

in 

Logits 

Odds 

Ratio 

Std. 

Error 

Means 

EC C 

Effect 

Size 

P-Value 

ELA 

Full sample 

2,458 

0.14 

NA 

0.04 

0.37 

0.23 

0.147 

<.0001 

assessment 

score 

Excluding 
missing outcome 
data 

2,140 

0.12 

NA 

0.03 

0.42 

0.30 

0.126 

.0004 


NOTES: The treatment group means are unadjusted probabilities; the comparison group means were computed based on the 
unadjusted treatment group means and estimated Early College effects. 
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APPENDIX F: SENSITIVITY ANALYSES 


Comparison of Impact Results That Included and Excluded 
Baseline Characteristics 

We included baseline characteristics in all analyses. As a sensitivity test, we also reran our 
main ITT model for three outcomes excluding baseline characteristics (gender, minority, low 
income, first generation, and prior test scores), and findings were similar to our inclusive 
model. See Exhibit F.10 for the comparison between models that included and excluded 
baseline characteristics as covariates. 


Exhibit F.10. Comparison of Impact Results That Included and Excluded Baseline Characteristics on 
High School Graduation, College Enrollment, and Degree Attainment 







Probabilities 



Outcome 

Analysis 

Sample 

Effect in 
Logits 

UCjCjS 

Ratio 

oia. 

Error 

EC 

C 

cnect 

Size 

P-Value 

High school 

Full sample 

0.39 

1.48 

0.15 

86.0% 

80.6% 

0.236 

.0103 

graduation 

Excluding 

baseline 

characteristics 

0.40 

1.50 

0.15 

86.0% 

80.4% 

0.244 

.0066 

Ever enrolled in 

Full sample 

0.50 

1.65 

0.11 

79.7% 

70.5% 

0.302 

<.0001 

college 

Excluding 

baseline 

characteristics 

0.49 

1.64 

0.11 

79.7% 

70.6% 

0.298 

<.0001 

Any 

Full sample 

2.69 

14.79 

0.23 

22.2% 

1.9% 

1.633 

<.0001 

postsecondary 

degree 

Excluding 

baseline 

characteristics 

2.59 

13.30 

0.22 

22.2% 

2.1% 

1.568 

<.0001 


NOTES: The treatment group probabilities are unadjusted probabilities; the comparison group probabilities were computed 
based on the unadjusted treatment group probabilities and estimated Early College effects. 
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APPENDIX G: COMPLIER EFFECTS ANALYSES 


Appendix G: Compiler Effects Analyses 

Given the presence of no-shows and crossovers in some sites, we supplemented the ITT 
effect analyses with complier effect analyses that estimate the impact of attending an Early 
College (as opposed to being offered admission) for lottery winners. The Early College 
effect on compliers (i.e., lottery winners who attended an Early College) was estimated using 
an instrumental variable (IV) approach, which is particularly well suited to estimating the 
effects of mediators (e.g., treatment receipt, or Early College attendance, in this case) in an 
experimental context (Angrist, Imbens, & Rubin, 1996; Gennetian, Morris, Bos, & Bloom, 
2005). An IV analysis typically involves two stages. During the first stage, an instrumental 
variable is used to obtain the predicted values of the mediator of interest. The predicted 
values, instead of the original values, of the mediator are then used in the second stage to 
predict the outcome. Important covariates are included in both stages to improve the 
precision of the IV estimates of the mediator effect. 

For the IV approach to generate a valid estimate of the causal effect of the mediator on the 
outcome, two conditions must be met. First, the instrument must co-vary with the mediator. 
All else being equal, the stronger the covariation, the more precise the IV estimates. 

Second, the instrument must be uncorrelated with the outcome when the mediator is taken 
into account. This condition is also known as the exclusion restriction assumption, which 
essentially assumes that the effect of the instrument on the outcome is completely mediated 
through the mediator and that no other causal paths exist between the instrument and the 
outcome. 

In the Early College impact study, treatment assignment is expected to affect, or co-vary 
with, Early College attendance, because students who won the lottery were more likely to 
attend an Early College than students who did not, as evidenced by the relatively low no- 
show and crossover rates in most sites. 60 It is difficult to hypothesize a path by which 
winning the lottery could improve student outcomes other than through Early College 
attendance (i.e., exclusion restriction). Lottery-based random assignment is thus a natural 
choice of instrument for Early College attendance in our case, as it meets both conditions for 
a valid IV analysis. The logic of the analyses of Early College effects on lottery winners who 
attended Early Colleges based on the IV approach is illustrated in Exhibit G.l. 


Exhibit G.l. Logic of Complier Effects Analysis Based on the Instrumental Variable (IV) Approach 



60 For the purpose of complier analyses, we defined Early College attendance based on whether a student attended an Early 
College (regardless of whether or not the Early College was in the study) during the first year of high school. 
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APPENDIX G: COMPLIER EFFECTS ANALYSES 


Specifically, our IV analyses of the Early College effects on lottery winners who attended 
Early Colleges proceeded in two stages. In the first stage, we obtained the predicted 
probability (in logits) of attending an Early College for each student based on a model similar 
to the one used for the ITT impact analysis (see Appendix D), except that the dependent 
variable for the first-stage equation was Early College attendance rather than a student 
outcome. Based on the results of the first-stage equation, we converted the predicted values 
of Early College attendance (in logit) for each student into the predicted probability of 
attendance that ranged between 0 and 1. The second-stage equation of the IV analyses 
also was similar to the ITT impact model, except that the treatment indicator variable in the 
ITT model was replaced with the predicted probability of attendance estimated during the 
first stage. 

Ideally, the two stages of an IV analysis should be estimated simultaneously in a single 
analysis to properly take into account the estimation error associated with the predicted 
probability of Early College attendance from the first stage. However, the IV analysis 
modules available in existing software programs cannot accommodate multilevel logit 
models for binary outcomes. Therefore, we carried out the two IV stages in two separate 
steps. This two-step IV approach may have somewhat underestimated the standard errors 
of the complier effect estimates, as it could not properly take into account the uncertainty in 
the predicted values of the Early College attendance. The size of the effect estimates, 
however, is not affected by the two-step approach. 

The IV estimates of complier effects of Early Colleges as well as the corresponding ITT 
effects are summarized in Exhibit G.2. As expected, the complier effect estimates were 
larger than the ITT effect estimates. For example, for the outcome of ever earning a 
postsecondary credential, the effect size for the complier effect was 2.229, as compared 
with an effect size of 1.633 for the ITT effect. The statistical significance of the complier 
effects, however, was generally consistent with the significance of the ITT effects. 
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APPENDIX G: COMPLIER EFFECTS ANALYSES 


Exhibit G.2. Estimates of Complier Effects and ITT Effects of Early Colleges on High School Graduation, 
Ever Enrolled in College, and Any Postsecondary Degree 


Outcome 

Analysis 

n 

Effect in 
Logits 

Odds 

Ratio 

Std. 

Error 

Probabilities 

EC C 

Effect 

Size 

P-V alue 

High School Graduation 

ITT effect 

2,458 

0.39 

1.48 

0.15 

86.0% 

80.6% 

0.236 

0.481 

.0103 

Complier effect 

2,458 

0.79 

2.21 

0.24 

89.2% 

78.9% 

.0009 

Ever Enrolled in College 

ITT effect 

2,458 

0.50 

1.65 

0.11 

79.7% 

70.5% 

0.302 

<.0001 

Complier effect 

2,458 

0.95 

2.58 

0.16 

83.3% 

66.0% 

0.574 

<.0001 

Any Postsecondary Degree 

ITT effect 

2,458 

2.69 

14.79 

0.23 

22.2% 

1.9% 

1.633 

<.0001 

Complier effect 

2,458 

3.68 

39.57 

0.31 

26.8% 

0.9% 

2.229 

<.0001 


NOTES: The treatment group probabilities are unadjusted probabilities; the comparison group probabilities were computed 
based on the unadjusted treatment group probabilities and estimated Early College effects. 
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